Looking For Software Developer Input...

I'm hoping a software developer or two (I know you guys are out there) can chime in on whether this makes sense or not.

As part of my brewing setup, I want to use an Arduino Mega with Ethernet to handle the gather of data (temperatures, volumes, etc) and then send that data to a server for processing and storage into a SQL database. The Arduino would probably also be running a PID routine on it for controlling my RIMS tube.

I would then like to run some type of web-enabled interface on the server that will allow me to both view the data and send commands back to the Arduino (i.e. change target temp). Ideally I would like the web interface to be multi-platform so that it works on both Internet Explorer and the iPad.

Technology Selection
My skills are really in the Microsoft space but more on the infrastructure side than development. I can work my way through small programming tasks and copy and paste with the best of them.

I would like to stick with the Arduino Mega as I have a decent grasp of working with it and there's a great amount of example code out there (including the BrewTroller code); I also own it already. I can see where the Raspberry Pi might make more sense in the future but the Arduino seems like it could be sufficient for now.

Here's where I was hoping for a developer's views:

• I was looking on running node.js (on an IIS server which is probably blasphemy) for the server-side and then using the websockets package on the Arduino to communicate to/from node.
  
• For client communication from IE and the iPad, I assumed this would also be websockets which I know is only supported in IE10 but I'm okay with that.
  
• For connectivity between node and the database, is JSON the appropriate technology?

If anyone has comments I would appreciate it.

Thanks!

Joe

What's the motivation for a separate server? If you're just looking for data presentation and a way to feed commands, it would be quite a bit simpler to just serve the HTML directly from the Arduino. I'm certainly not trying to convince you out of anything, but I want to make sure I understand why you're going this route before I make any suggestions.

MalFet said:

What's the motivation for a separate server? If you're just looking for data presentation and a way to feed commands, it would be quite a bit simpler to just serve the HTML directly from the Arduino. I'm certainly not trying to convince you out of anything, but I want to make sure I understand why you're going this route before I make any suggestions.

Click to expand...

I suppose I just assumed that a separate server would provide me more flexibility in creating a "pretty" interface and I wanted to store the data externally in a database anyhow for analysis. I figured I would eventually work towards a more robust interface that would include something like upload of a BeerXML file for inputs that would be used for parameters on the Arduino. Also, I didn't see any capabilities to provide asynchronous communication from the Arduino itself.

My thought was that the Arduino would be humming along doing it's data stuff and I could have client devices that talk to a web page consuming the data and sending back commands if necessary. Temperature updates on the web page would be fairly realtime whereas data would only be submitted to the database on set intervals.

One other piece I was looking at was something like Flot for graphing the data and providing realtime feedback.

I'm certainly open to suggestions...

jpalarchio said:

I suppose I just assumed that a separate server would provide me more flexibility in creating a "pretty" interface and I wanted to store the data externally in a database anyhow for analysis. I figured I would eventually work towards a more robust interface that would include something like upload of a BeerXML file for inputs that would be used for parameters on the Arduino. Also, I didn't see any capabilities to provide asynchronous communication from the Arduino itself.

My thought was that the Arduino would be humming along doing it's data stuff and I could have client devices that talk to a web page consuming the data and sending back commands if necessary. Temperature updates on the web page would be fairly realtime whereas data would only be submitted to the database on set intervals.

One other piece I was looking at was something like Flot for graphing the data and providing realtime feedback.

I'm certainly open to suggestions...

Click to expand...

Arduino programs are single threaded, so there's no true asynchronous communication. That said, if you throw in a line in your main loop to sniff for a waiting client connection it will functionally behave that way.

I know this is an old thread, but I figured I might chime in. I think saving to SQL might be an over complication. Writing the file as a text (CSV file) seems like a simple solution, you can then parse that data into an excel, sql, or xml file on your PC. As far as getting the info form the Arduino you have a few options. You add a local storage device, then physically remove it and copy it to your PC, or you can use processing and some sort of serial communication to send the data to your PC (or mac) and save the data there.

jpalarchio said:

I'm hoping a software developer or two (I know you guys are out there) can chime in on whether this makes sense or not.

As part of my brewing setup, I want to use an Arduino Mega with Ethernet to handle the gather of data (temperatures, volumes, etc) and then send that data to a server for processing and storage into a SQL database. The Arduino would probably also be running a PID routine on it for controlling my RIMS tube.

I would then like to run some type of web-enabled interface on the server that will allow me to both view the data and send commands back to the Arduino (i.e. change target temp). Ideally I would like the web interface to be multi-platform so that it works on both Internet Explorer and the iPad.

Technology Selection
My skills are really in the Microsoft space but more on the infrastructure side than development. I can work my way through small programming tasks and copy and paste with the best of them.

I would like to stick with the Arduino Mega as I have a decent grasp of working with it and there's a great amount of example code out there (including the BrewTroller code); I also own it already. I can see where the Raspberry Pi might make more sense in the future but the Arduino seems like it could be sufficient for now.

Here's where I was hoping for a developer's views:

• I was looking on running node.js (on an IIS server which is probably blasphemy) for the server-side and then using the websockets package on the Arduino to communicate to/from node.
  
• For client communication from IE and the iPad, I assumed this would also be websockets which I know is only supported in IE10 but I'm okay with that.
  
• For connectivity between node and the database, is JSON the appropriate technology?

If anyone has comments I would appreciate it.

Thanks!

Joe

Click to expand...

I like the idea of sending the data to a SQL database (most likely mysql) residing on a server, which can just be a computer on your local net running Apache. I would recommend using AJAX so that your browser is being updated with the most recent data without a refresh; however, I'm not sure that this works with Apple's browser, Safari. This will only be near real-time since you have to wait for the Arduino to update the database, but it's still cool.

I would just use the arduino for control of your rims and build in a mechanism to read/write current process variables via TCP. Then you could write an app running on a PC to get the data from the Arduino in some configurable interval (say every 20 seconds). The app could log this data to your DB of choice. From there the possibilities of presenting the data are infinite.

mdgagne said:

I would just use the arduino for control of your rims and build in a mechanism to read/write current process variables via TCP. Then you could write an app running on a PC to get the data from the Arduino in some configurable interval (say every 20 seconds). The app could log this data to your DB of choice. From there the possibilities of presenting the data are infinite.

Click to expand...

This is along the lines of what I'm thinking. The PID algorithm running on the Arduino is going to need more timely updates from the temperature sensors than I need logged. I don't need 5 billion data points logged during a brew session.

I was trying to avoid writing any type of client/server application and instead leverage existing technologies to make it sort of a web app; still doing a lot of reading there.

The single-threaded nature of the Arduino seems like it might squash some of my initial thoughts.

What I need to do on the Arduino is:

Listen For Action Command From Server (via sockets?) -> Get Current Temperatures -> Execute PID -> Upload Data To Server (via sockets?) -> REPEAT

What's not clear to me within this loop is when I'm waiting for actions from the server, how long do I wait and what does the server do as far as sending when I'm at a point where the Arduino is not listening.

jpalarchio said:

This is along the lines of what I'm thinking. The PID algorithm running on the Arduino is going to need more timely updates from the temperature sensors than I need logged. I don't need 5 billion data points logged during a brew session.

I was trying to avoid writing any type of client/server application and instead leverage existing technologies to make it sort of a web app; still doing a lot of reading there.

The single-threaded nature of the Arduino seems like it might squash some of my initial thoughts.

What I need to do on the Arduino is:

Listen For Action Command From Server (via sockets?) -> Get Current Temperatures -> Execute PID -> Upload Data To Server (via sockets?) -> REPEAT

What's not clear to me within this loop is when I'm waiting for actions from the server, how long do I wait and what does the server do as far as sending when I'm at a point where the Arduino is not listening.

Click to expand...

I have not done much work with Arduinos boards, but I have done some development with Atmel AVRs which are the microprocessors used on all the Arduinos. The rs-232 port works with an interrupt service routine so you shouldn't have to worry about waiting for requests if the Ethernet stuff works the same. Even though you're dealing with a single threaded application, interrupts work great for dealing with I/O in a seemingly asynchronous fashion, just remember that anytime the Arduino has to service an interrupt it is not executing the main process loop.

jpalarchio said:

This is along the lines of what I'm thinking. The PID algorithm running on the Arduino is going to need more timely updates from the temperature sensors than I need logged. I don't need 5 billion data points logged during a brew session.

I was trying to avoid writing any type of client/server application and instead leverage existing technologies to make it sort of a web app; still doing a lot of reading there.

The single-threaded nature of the Arduino seems like it might squash some of my initial thoughts.

What I need to do on the Arduino is:

Listen For Action Command From Server (via sockets?) -> Get Current Temperatures -> Execute PID -> Upload Data To Server (via sockets?) -> REPEAT

What's not clear to me within this loop is when I'm waiting for actions from the server, how long do I wait and what does the server do as far as sending when I'm at a point where the Arduino is not listening.

Click to expand...

Like I said before, far and away the easiest way to do this is to simply poll for client requests each time through your main loop. This isn't true asynchronous operation, but it would be essentially indistinguishable from it for your purposes.

Otherwise, you can use hardware interrupts like mdgagne mentioned, though that gets a bit more complex on the arduino.

jbrewkeggin said:

I would recommend using AJAX so that your browser is being updated with the most recent data without a refresh; however, I'm not sure that this works with Apple's browser, Safari.

Click to expand...

Ajax works fine with Safari, and has for years.

jpalarchio said:

This is along the lines of what I'm thinking. The PID algorithm running on the Arduino is going to need more timely updates from the temperature sensors than I need logged. I don't need 5 billion data points logged during a brew session.

I was trying to avoid writing any type of client/server application and instead leverage existing technologies to make it sort of a web app; still doing a lot of reading there.

The single-threaded nature of the Arduino seems like it might squash some of my initial thoughts.

What I need to do on the Arduino is:

Listen For Action Command From Server (via sockets?) -> Get Current Temperatures -> Execute PID -> Upload Data To Server (via sockets?) -> REPEAT

What's not clear to me within this loop is when I'm waiting for actions from the server, how long do I wait and what does the server do as far as sending when I'm at a point where the Arduino is not listening.

Click to expand...

What you are trying to do is typically done via socket io select on the server side (like with apache httpd for example)...

I don't know if the Arduino supports io select but you can build a loop that does similar in a serial nature. Basically you will listen on a socket (a tcp port) in a while 1 loop, when a connection is made you read data from the socket and process accordingly (you would have to com up with some type of cmd structure or protocol)

You need to decide who the client is in the conversation: IE who starts the conversation.

So here is a verbal representation of what I like you wan to do.

Arduino = Server

-> Client connects a port on server and sends a command
<- Server reads command, process, returns the process output to client and closes the connection
-> Client receives the data returned and logs it to a database

So something like this

-> open socket port 1234, write Hello World
<- read from socket port 1234, read Hello World , write World Hello
-> read from socket World Hello, write to database World Hello.

Probably a little late to throw this out there but if you have a Microsoft/.NET background, Netduino will be much easier since it runs .net micro

As for your setup, you want separation of concerns in terms of the server and the controller so your controller can run all of the time and the server becomes the 'guiding hand' per se. That way your controller is 'disconnected' from the server so it can run regardless of any issues on the server-side.

I would architect it like this:

Arduino Mega
-- Runs brewery based on some type of config source that defines variables for the brewing routine (times, target temps, etc). The source needs to be local, read/writable via Ardunio and the server. Likely a USB stick on the Ardunio board?
-- Stick some folders on the USB stick; /logs (write logging data with Ardunio), /currentBrew (store your current brewery config here which Arduino will use to run the brewery)

Server
-- Reads data from the /logs folder -- so it'll show temp data, times, etc.
-- Able to edit the configuration in the /currentBrew folder


Now Ardunio needs the ability to detect changes to the /currentBrew folder and apply those changes. If you want some type of validation that changes were processed by Ardunio correctly, you can use a /serverCommands folder to store commands for Ardunio to execute. As Ardunio iterates through the commands and successfully executes them, just remove the line from the file. When the line is gone, assume the command was processed.

While not an elegant architecture, Ardunio is very compartmentalized and never has to do any network functions at all other than open up the USB stick to the server. Flat text-file config and log files will make debugging a breeze. And your server doesn't have to do anything fancy but read a log file and write to a text file. Use jQuery mobile for the server UI and you'll have a mobile-ready 'brewery controller'.



Just a very rough overview. If you want some more input, feel free to contact me.