Size and depth of structure issue
fogled@mizzou
h4x354x0r
I've been revamping some of my code, and creating bigger nested structures to hold all my device information. At this point, I've got a structure with up to 4 nested sub-structure levels that adds about 200K to the TKN file on upload.
When I have this structure defined, no code will run on my controller. When I comment the structure out, code will run OK. I seem to be breaking some limit somewhere, but... I don't know what it is.
When I have this structure defined, no code will run on my controller. When I comment the structure out, code will run OK. I seem to be breaking some limit somewhere, but... I don't know what it is.
Comments
-
The max depth of an array is 5 dimension I beleive. Perhaps you've gone over that limit in your structures.
-
I'm pretty sure I'm only 4 deep. Here's my actual structure code at the moment:
structure deviceip { char name[32] char ipaddr[20] integer ipport integer iptype dev devid } structure msgs { char msgdate[19] char msgtext[80] } structure lamps { integer pwr char hr[5] } structure temps { char val[5] } structure valmap { char str[32] } structure paneldat { char name[32] integer tab dev devid char tpbuff[100] char tpinput[100] } structure displaysettings { char name[32] integer pwr integer vmute integer feed integer follows integer status integer swtid //VGA Switch ID integer vport //VGA Port Number integer type integer commtype deviceip ipcomm[1] char sercomm[64] dev serdevid[1] integer chan integer amute integer volcatv integer vol integer volset integer tmp integer projerr integer errorcount char text[32] char buffout[64] //holds buffer of integer commands as string data for fifo operations char buffin[1024] integer lastcmd integer msgidx msgs currmsg[1] msgs prevmsg[128] lamps lamp[1] temps temp[1] integer lamplimlevel } structure audchans { char name[20] char id[1] char chan[1] integer val integer mute integer volset } structure audiodata { char name[32] integer type integer commtype deviceip ipcomm[1] char sercomm[64] dev serdevid char buffin[256] //Extron Mp121 char buffout[256] //Extron MP121 audchans ch[20] } structure lightchans { char name[20] char chan[1] integer volset integer val integer isgang } structure lightingdata { char name[32] integer type integer commtype deviceip ipcomm[1] char sercomm[64] dev serdevid[1] char buffin[256] //Extron char buffout[256] //Extron lightchans ch[20] } structure inputs { char name[32] integer swtid //VGA Switch ID integer vport //VGA Switch Video Port (nornally Aud and Vid) integer aport //VGA Switch Audio Port (Use separate Aud/Vid for oddball stuff) integer follows } structure switchdata { char name[32] integer type char sercomm[64] dev devid char buffin[512] char buffout[512] } structure helpdata { integer helpbtns[32] char buffin[256] char buffout[1024] } structure room { char name[32] char dbid[12] integer status paneldat panels[1] displaysettings d[8] audiodata audio[1] inputs in[8] lightingdata lights[1] switchdata swt[2] helpdata help[1] // what else in the room? char helpbuffin[256] //Help/Log gateway char helpbuffout[1024] //Help/Log Gateway }
...and then in Define Start, I have: PERSISTENT roomdata rm[20]
Actually, you may be right because I'm using DEV way down in rm[x].d[x].ipcomm[x].devid - which is itself a DEV, but that's still just 4 deep, not 5 deep. I can't spot anything that goes more than 4 deep, can anyone else? Is there anything else in here that's a no-no?
Thanks, -
It looks like the total structure size was just too big. In the DisplaySettings structure, I had a "msgs" sub-structure with 128 slots, and that sub-structure had an 80-char container.
I backed the msgs structure call off to 10 instead of 128, and the system works OK. -
Telnet into your master and type in "show mem" . That will tell you how much memory of each type is being used. I'm willing to be you ran out of non-volatile memory. There is far less non-volatile available, and variable declarations default to it, so unless you specifically declared it volatile, it's a good bet it ate up all your non-volatile memory. If you need it to be persistent, write it to a file and read it on startup so it can remain volatile.
-
One thing I found with Netlinx that wasn't intuitive is that nested structures are sloooowwww. When I was writing a Netlinx Sonos module, I couldn't figure out why simply looping through a nested structure and retrieving data to send to a touch panel took so long. I converted it all to arrays, and was shocked at how the same code ran so much faster. I think that memory accesses to structures mustn't be optimized, and read/writes are much slower. So now I only use structures without nesting or very small nested structures. It would be great to be able to use them as objects of arbitrary size but it doesn't seem to work well in implementation.
Paul -
One thing I found with Netlinx that wasn't intuitive is that nested structures are sloooowwww.
So far, I haven't run into this. I'm walking an x[20].y[10].data nested structure (as big as it's going to get) with nested FOR loops , pulling data as needed, and it does the whole thing in a fraction of a second. I'll keep my eyes peeled for slowness, though.
I totally forgot I could telnet in and do a Show Mem though; thanks for the reminder. ;-) -
if you are running through a LARGE struct array, the difference becomes clear, especially when looping many times for touchscreen feedback, its a factor of 4 times difference at least.
-
if you are running through a LARGE struct array...
Mine *WAS* large, until I put it on a diet. But I don't actually "walk" the structures anywhere but startup when I need to open all the IP ports the system makes connections to. Otherwise, everything is a direct call to a specific cell as needed. I'm not even using the message trapping part of my structure right now, it was just on the slate for implementation.
The "Program" part where all the feeback is, though... the system walks that all the time anyway, although the calls in the feedback are mostly only 2 levels deep, occasionally 3 (audio channel mute state). There should be a slowdown there, shouldn't there?
However, I doubt I'll see any noticeable slowdown, just because my structure isn't really *that* big.
Still interesting to know structures are much slower than arrays though. Not surprising, just... interesting. -
A side note here, I actually *can* detect that the nested structure is way slower than non-nested structures or arrays. But the only reason I could is because of a very poor command execution order in a block of code. So far, the slowness doesn't actually impact the system operation (once I fix my execution order).
-
I don't know how the AMX compiler handles structures, but if you have an array of these structures, it allocates ram for ALL fields, even if you are only using one or two values from each array. Also, if you have buttons/channels/etc declared in your system, the system allocates RAM to keep track of ALL channels up to the number you are declaring. So if you have
DEFINE_VARIABLE
INTEGER testChan = 255
and then
Button_Event[dvtpTestChan, testChan]
then your netlinx system allocates memory for all channels 1-254. in addition to 255.
Is there a reason for you to have such a monster struct? I generally like to have smaller structs, as they make the lookup process faster, and I can break them out into separate AXI files to make maintenance easier. -
The reason for the monster struct is because I'm a database developer too, and that's more or less the way the data ends up being normalized. I use one controller for multiple rooms - I've put an arbitrary limit of 20 just to be on the safe side - and each room has multiple devices, and each device can have multiple settings that need tracked. It ends up being a 3-4 level deep data structure, but it makes complete sense for the rest of my programming and has made it that much easier to do the rest of my programming.
I'm aware of the waste in memory allocation, it is what it is. I buy the high-end controllers with more memory so I can be lazy an more consistent about my system design and programming. The only reason I killed this one was because I had a defined a huge error message storage system to get around the fact that the controllers lose all their console messages on power loss.
But I don't really need that so much, because I've got the controllers reporting back to a central database system which stores the messages (kind of a homegrown RMS but without all the unnecessary crap). So, I just scaled back the size of that part, and got within the memory bounds.
RANT ON
At this point, I've started co-developing another system to replace AMX controllers, built on all open-source stuff: Postgre for data storage, Python for core logic processing, and PHP to serve UI's. Then, IP-serial devices for the crap that doesn't have IP-based communications. The design of my structures in AMX mirrors the structure of my real database I'm developing.
Quite frankly, I'm pissed at the way AMX has re-structured their technical support. I'm not a dealer, but I deal with dealer-sized deployments and equipment counts, and where a quick phone call used to get me what I needed, now it's a huge hassle to get anything out of AMX. As a result, I'm accelerating my drive to move away from vendor-specific solutions. I put myself at the mercy of AMX, and they cut me at the knees for it. I won't put up with that any longer than I have to.
And don't get me started on the AMX University bit. I tried to get one of my co-workers registered in that system; we kept filling out and submitting the forms, and we never hear anything back. I buy tons of AMX stuff, but it never gets credited to my account. From my perspective, it's nothing but a hokey PR disaster that's left a very bad taste in my mouth. I gave up on it.
I still need AMX right now, because I don't have anything better to take it's place yet, but AMX's days are numbered on my campus. In the meantime, I'll push the AMX controllers to their limit.
RANT OFF -
fogled@mizzou wrote: »RANT ON
Quite frankly, I'm pissed at the way AMX has re-structured their technical support.
RANT OFF
If you don't like AMX support, I doubt you will like PHP, Postgre, and Python support much either.
Paul -
The difference is, I know I'm on my own, but everything is out in the open and I can find out whatever I need. I've dealt with these open source tools for years; I can provide my own support for them more quickly and effectively than AMX can provide support for their proprietary products. Resolving AMX issues used to be easier and faster than researching open source, but for me, since the changes, not anymore.
The difference is, not being at any one person's or vendor's mercy. Our campus has dozens of very competent programmers who know these open source tools even better than I do. Our campus has only one certified AMX programmer - me - and only two in our entire town, and I can't even directly get help from AMX anymore.
The difference is, the open source software runs great on redundant virtual servers on our production floor, whose cost has already been completely amortized within our organization. AMX hardware very reliable, but is poor at redundancy, and quite expensive in comparison.
Sorry to sound so pissy about it. AMX can make their decisions, and I can make mine. Customer service is extremely important to me, and my perception is that AMX changed formerly great customer service into currently horrible customer service. I realize not everyone has experienced the same; I'm just one of those outliers that fell through the cracks. But as one of the few that has been cut off from good support, I have to react to the situation I find myself in. My reaction is to walk away in as orderly a fashion as I can. -
Let me say, AMX for the most part I am happy with, but I do understand your plight. Let me make a suggestion though.fogled@mizzou wrote: »At this point, I've started co-developing another system to replace AMX controllers, built on all open-source stuff: Postgre for data storage, Python for core logic processing, and PHP to serve UI's. Then, IP-serial devices for the crap that doesn't have IP-based communications. The design of my structures in AMX mirrors the structure of my real database I'm developing.
I would either use
http://www.jr.com/startech/pe/TCH_PEX8S952/
or
http://www.jr.com/startech/pe/TCH_ICUSB23216F/
I can confirm both work quite well, instead of ip > serial, sometimes having an issue when the tcp/ip stack is waiting for more data to form a filled packet instead of sending with no delay.
I made a .net based system recently, but, decided to not pull the trigger. FWIW, look at the patent document and wireshark captures and watch the panel > master comms and you can make your panels play nice with you new system. -
I've been using MOXA N-port units, and haven't had any problems with them so far. From a deployment perspective, those seem a lot more modular than having to install a card in a PC, which changes fairly frequently and isn't anywhere near what I would consider control-system level reliable. Most room PC's on campus are targeted to be replaced by thin clients anyway, which don't have card slots at all.
In the best of worlds, I wouldn't even need to use serial. I don't drive serial to projectors anymore; all new installations are direct IP connections. All the higher-end audio DSP's are IP-based too. For really simple rooms where we just run audio control through the projector, I just drop in a touchpanel and a projector, run a network cable to each, enable them in the central controller, and I'm done. It's simple and relatively inexpensive, compared to controllers in each room and serial everywhere.
For the slightly more complex rooms, I drop a MOXA in to drive serial to a switch and a low-end audio control device. It's mostly just video switches, low-end audio controllers, and lighting systems that still require serial. Still modular and less expensive than dedicated controllers.
For the big rooms, I still buck up for a dedicated controller, but I'm getting really close to being able to drive even the biggest systems off a controller on my production floor with only IP-based connections. That's part of what my "monster" structure is all about. I'm hoping this summer will be the last time I wind up installing a dedicated controller for a single room of any size.
The idea with the server-based open source system is to completely ditch proprietary hardware, including touchpanels. Instead, I'll put the controls on smartphones and tablets for user convenience, and fixed touchscreen kiosk units in rooms. That's all commodity hardware, not vendor locked, and far less expensive.
This is what AMX is competing against. I can understand they need to cut costs, but cutting customer service to people like me is really just shooting themselves in the foot. My open-source system would still just be an idea if I hadn't been kicked out of being able to call their tech support directly, and take a month-long round-trip through a 3rd-party vendor just to get a piece of equipment repaired. But now, I've already got the project underway, and a simple test system in operation.
This is the last post I'm going to make on this thread. I've spoken my piece. Thanks for listening and understanding.
Categories
- All Categories
- 2.5K AMX General Discussion
- 922 AMX Technical Discussion
- 514 AMX Hardware
- 502 AMX Control Products
- 3 AMX Video Distribution Products
- 9 AMX Networked AV (SVSI) Products
- AMX Workspace & Collaboration Products
- 3.4K AMX Software
- 151 AMX Resource Management Suite Software
- 386 AMX Design Tools
- 2.4K NetLinx Studio
- 135 Duet/Cafe Duet
- 248 NetLinx Modules & Duet Modules
- 57 AMX RPM Forum
- 228 MODPEDIA - The Public Repository of Modules for Everyone
- 943 AMX Specialty Forums
- 2.6K AMXForums Archive
- 2.6K AMXForums Archive Threads
- 1.5K AMX Hardware
- 432 AMX Applications and Solutions
- 249 Residential Forum
- 182 Tips and Tricks
- 146 AMX Website/Forums