A $5 Programmable Tiny Webserver The Size of a Jump Drive

• Download source code - 27.3 KB

Introduction

The future is here! We now have astoundingly cheap and tiny networkable 32-bit CPUs, and they are starting to become ubiquitous. In this article, we delve into the ESP-01 module and program a small webserver on it.

Update: Added support for embedding content into a filesystem in flash memory

Update 2: Added a tool for gzipping and ungzipping directories for use with the webserver

Prerequisites

You'll need an ESP-01 module and a CH340 based USB to ESP8266 adapter. Both of these are pictured above.

You'll need the Arduino IDE. You must go to File|Preferences and add the following to the Additional Board Managers text box:

• http://arduino.esp8266.com/stable/package_esp8266com_index.json

If there is already a URL present, delimit them with commas.

Ensure under Tools|Board: it reads "Generic ESP8266 Module".

Plug the ESP8266 module into the USB adapter like the one shown above such that the ESP8266 is over the USB adapter rather than hanging off the end. If you look at it from the side, it should form a "U" of sorts. That's for this adaptor. If yours is different, it may vary. Just be careful or you can damage your device.

If you want to use the technique outlined later wherein we use some of the flash storage as a filesystem, you'll need to do the following:

First, go get yourself the latest ESP8266FS zip file here.

If you're on Linux, you'll want to get the ESP8266FS folder in the zip, and find your Arduino application directory. That should be off your home directory. Mine is ~/arduino-1.8.13. Do not get it mixed up with ~/Arduino. Under there, there is a tools folder and that's where you want your ESP8266FS folder to go.

I've never done it on Windows, but this is how you do it: You'll need the ESP8266FS folder from the zip. Find your program directory. It is probably something like C:\Program Files (x86)\arduino-1.8.13. Inside, there is a tools folder. That is where the ESP8266FS folder needs to go.

Either way, you'll need to restart the Arduino IDE. You'll know it took if you now have a Tools|ESP8266 Sketch Data Upload option.

Conceptualizing this Mess

The ESP8266 based ESP-01 module contains a WiFi transceiver and a 32-bit processor operating at 80MHz powered off of a modest 3.3 volt power source - often USB. Its main I/O facility aside from WiFi is a single serial UART. As is often the case with my code, that serial UART will be where all of the debug and status messages get sent. With the CH340 based USB adapter, it will expose that UART as a virtual COM port. With this scenario, you can use that COM port to monitor what the device is doing.

We'll be creating a web server which I based on some of the example code. This webserver will automatically connect to the configured SSID and then expose itself under the URL http://test.local. The webserver simply displays an image and some text. The test.local domain is exposed using Multicast DNS.

Since there is often no storage, much less a filesystem, all of our content must be embedded in the source code itself. It is possible to use a filesystem, and when we do that, we'll be using a slightly different technique to serve the pages.

Coding this Mess

You'll have to make sure your USB adapter is set to "program" rather than "uart" whenever you go to upload code. You must then remove the USB adapter, switch it to "uart" and then reinsert it in the USB slot in order to run the code.

Without an Embedded Filesystem

Here is the code for the server. Note that I've truncated the main page and the image data for the purposes of displaying the content here without massive line wrapping. Do not copy and paste this code because it won't work due to that. Use the download at the top of the article to get the full .ino file included as a solution item.

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include <ESP8266mDNS.h>

#ifndef STASSID
// BEGIN configuration properties
#define STASSID "myssid"
#define STAPSK  "mypassword"
#define STAHOSTNAME "test"
// END configuration properties
#endif

const char* ssid = STASSID;
const char* password = STAPSK;

// create the server with the port to listen on
ESP8266WebServer server(80);

// the LED pin is 13
const int led = 13;

// handles requests to /
void handleRoot() {
  digitalWrite(led, 1);
  server.send(200, "text/html", "<!DOCTYPE html>\r\n<html ...");
  digitalWrite(led, 0);
}

// handles when content is not found
void handleNotFound() {
  digitalWrite(led, 1);
  String message = "File Not Found\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET) ? "GET" : "POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i = 0; i < server.args(); i++) {
    message += " " + server.argName(i) + ": " + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  digitalWrite(led, 0);
}

void setup(void) {
  pinMode(led, OUTPUT);
  // turn off the LED
  digitalWrite(led, LOW);
  // initialize the serial port
  Serial.begin(115200);
  // set up the wifi
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  // display the connection info
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("Host name: ");
  Serial.print(STAHOSTNAME);
  Serial.println(".local");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  // start the multicast DNS publishing
  if (MDNS.begin(STAHOSTNAME)) {
    Serial.println("MDNS responder started");
  }

  // install the handlers
  server.on("/", handleRoot);

  server.on("/inline", []() {
    server.send(200, "text/plain", "this works as well");
  });

  server.on("/test.jpg", []() {
    static const uint8_t img[] PROGMEM = 
      { 255, 216, 255, ... }
    ;
    server.send(200, "image/jpg", img, sizeof(img));
  });

  server.onNotFound(handleNotFound);

  // start the server
  server.begin();
  Serial.println("HTTP server started");
}

void loop(void) {
  // handle the request
  server.handleClient();
  MDNS.update();
}

I've included a tool for generating the content strings and byte arrays. It's called file2c and it is a command line utility I've attached to the project. You can use this tool like I did, to generate the content to pass to server.send() whether it's text or binary. It can generate C strings or C byte array initializers from input files. This way, you can edit an HTML page and then convert it to a C string when you're done in order to inject it into the above code. Running it is like this:

file2c mydoc.html 

or for a binary file like an image:

file2c myimage.jpg /binary

You can then copy the resulting output into your code.

With an Embedded Filesystem

Doing this requires a slightly different technique wherein we use the built in flash memory to store content and serve it from there. The filesystem used for the flash memory is called SPIFFS. We'll be doing it only when there isn't already a handler for the requested path. Basically, it intercepts before it would otherwise go to a 404 and if a file exists it will serve it instead. That way, any existing handlers can still work.

To do it, we need to add a couple of supporting functions to the webserver code presented above.

Note that I got this code from the article here.

First, we need to add a header to our code for the filesystem API:

#include <FS.h>

Next, we need to be able to associate a MIME content-type and a file extension, which is what the following method does:

String getContentType(String filename){
  if(filename.endsWith(".htm")) return "text/html";
  else if(filename.endsWith(".html")) return "text/html";
  else if(filename.endsWith(".css")) return "text/css";
  else if(filename.endsWith(".js")) return "application/javascript";
  else if(filename.endsWith(".png")) return "image/png";
  else if(filename.endsWith(".gif")) return "image/gif";
  else if(filename.endsWith(".jpg")) return "image/jpeg";
  else if(filename.endsWith(".ico")) return "image/x-icon";
  else if(filename.endsWith(".xml")) return "text/xml";
  else if(filename.endsWith(".pdf")) return "application/x-pdf";
  else if(filename.endsWith(".zip")) return "application/x-zip";
  else if(filename.endsWith(".gz")) return "application/x-gzip";
  return "text/plain";
}

Go ahead and add types as you need them. The other thing you'll need to do is add a method to deal sending a file to the client:

// send the right file to the client (if it exists)
bool handleFileRead(String path){  
  Serial.println("handleFileRead: " + path);
  // If a folder is requested, send the index file
  if(path.endsWith("/")) path += "index.html";           
  // Get the MIME type
  String contentType = getContentType(path);             
  String pathWithGz = path + ".gz";
  // If the file exists, either as a compressed archive, or normal
  if(SPIFFS.exists(pathWithGz) || SPIFFS.exists(path)){  
    // If there's a compressed version available
    // Use the compressed version           
    if(SPIFFS.exists(pathWithGz))                          
      path += ".gz";                                  
    // Open the file
    File file = SPIFFS.open(path, "r");                    
    // Send it to the client    
    size_t sent = server.streamFile(file, contentType);    
    // Close the file
    file.close();                                          
    Serial.println(String("\tSent file: ") + path);
    return true;
  }
  Serial.println(String("\tFile Not Found: ") + path);
  // If the file doesn't exist, return false
  return false;                                          
}

Notice how we have special handling for .gz files. This is so we can gzip our content to save precious flash space and a little bit of bandwidth. Basically, we store our content as like foo.html.gz or bar.jpg.gz and serve it that way. The browser will know how to display it.

Next, we need to update our handler where we'd normally just send a 404. Replace the handleNotFound() method with this slightly different code:

// handles when content is not found
void handleNotFound() {
  digitalWrite(led, 1);
  // If the client requests any URI
  if (handleFileRead(server.uri())) // send it if it exists
     return;
  String message = "File Not Found\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET) ? "GET" : "POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i = 0; i < server.args(); i++) {
    message += " " + server.argName(i) + ": " + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  digitalWrite(led, 0);
}

I put the change in bold.

Now we need to remove the following lines from the existing code:

// install the handlers
server.on("/", handleRoot);

server.on("/inline", []() {
 server.send(200, "text/plain", "this works as well");
});

server.on("/test.jpg", []() {
  static const uint8_t img[] PROGMEM = 
    { 255, 216, 255, ... }
  ;
  server.send(200, "image/jpg", img, sizeof(img));
});

and replace them with:

SPIFFS.begin(); 

in order to initialize the filesystem.

Now remove the handleRoot() method since we don't need it anymore.

Next, you must place all of your content to serve under the folder for your sketch in a folder called "data", so for example, if your sketch directory is ~/projects/myweb, your content needs to go under ~/projects/myweb/data.

At this point, you might consider gzipping each file to save space and a little bandwidth, but mostly space, since there's not much and so it's at a premium. You can use the gzdir utility to do so. For example, if we were under your sketch directory with gzdir in your PATH somewhere we would do this:

gzdir data

That should gzip each file in your data and delete the originals. You can reverse the process using the /decompress option like this:

gzdir data /decompress

Finally, once you're done uploading the code to the ESP8266, pull the ESP8266 assembly out of the USB socket and plug it back in to reset it so it can take another flash. Then you must use Tools|ESP8266 Sketch Data Upload to flash your files to the device. Be aware of errors. It's very easy to run out of space with the built in 1MB of flash.

Deploying

Once you're done programming your little webserver, you can power it without the USB stick if you want. It simply requires 3.3 vdc wired to the VCC, the same wired EN aka CHG, and then the GND connected to the ground or negative terminal.

Future Directions

You can potentially use the serial port to communicate with another board, say you had an Arduino or some other I/O board connected to it via serial. You can get the I/O using a technique nearly identical to the one outlined in this article. That way, you could wire sensors up to it and report their status using a dynamic web page for example. Another direction might be to wire up an SD card to the SPI interface so that you can have a filesystem and make it more of a real webserver.

There's also an SPI interface on the ESP-01 for communication but I know nothing about how to interface with it in software - yet!

History

• 3^rd November, 2020 - Initial submission
• 4^th November, 2020 - Update to include tiny filesystem
• 4^th November, 2020 - Added tool to gzip files