A not well documented Mapserver feature for Hillshade

If you have a hillshade GRID(or exported tiff) and DEM you can build a nice color hillshape map with Mapserver...  It's not really simple and well documented, but it's possible. I think this trick was add in Mapserver in 'experimental' mode in the trunk and gets released, but never realy discuss in the mailling list Mapserver-dev. To build this kind of service, we must have a DEM and Hillshade. It's not necessary to have a GRID format for this trick. A simple black and white tif format is fine. First you must know the pixel range to process in your DEM. You need it to let GDAL reader a pre-scale from incoming raster data to get the pixel range to process. You can easely use the gdalinfo utilities. ... Upper Left (-1051654.551, 2174097.741) ( 88d52'54.70"W, 62d14'24.12"N) Lower Left (-1051654.551, -201102.259) ( 80d59'1.05"W, 41d23'0.63"N) Upper Right ( 1359945.449, 2174097.741) ( 42d36'0.34"W, 61d21'52.09"N) Lower Right ( 1359945.449, -201102.259) ( 52d27'53.73"W, 40d50'27.53"N) Center ( 154145.449, 986497.741) ( 66d11'34.47"W, 52d52'40.66"N) Band 1 Block=256x256 Type=Int16, ColorInterp=Gray Min=-10.000 Max=706.000 Minimum=-32768.000, Maximum=1890.000, Mean=-185.108, StdDev=3252.745 ... In the example I have in Band 1, a min elevation of -10 to a max of 706. This resulte in the mapfile with PROCESSING "SCALE=-10,706". Then, you have to process the DEM dynamically with a COLORRANGE in TRANSPARENCY mode and overlay it on a hillshade. To get the DEM and the process hillshade in a single layer in your map, simply GROUP them in your mapfile. Example in the mapfile GROUP on "Ombrage_bleu_250K"...

...
LAYER
    NAME "ombre250k_blue"
    GROUP "Ombrage_blue_250K"
    DATA "mne250k/ombre250k.tif"
    TYPE RASTER
    MAXSCALE 4000000
    MINSCALE 1
    METADATA
        "wms_group_title"       "Ombrage blue 250K"
        "wms_name"              "ombre250k_blue"
        "wms_title"             "ombre250k blue"
        INCLUDE                 "../include/ec/ec_meta_layer.map"
    END
    PROJECTION
         "init=epsg:32198"
    END
END
LAYER
    NAME "mne250k_blue"
    GROUP "Ombrage_blue_250K"
    DATA "mne250k/mne250k.tif"
    TRANSPARENCY 30
    TYPE RASTER
    PROCESSING "SCALE=-1,706"
    METADATA
        "wms_group_title"       "Ombrage bblue 250K"
        "wms_name"              "mne250k_blue"
        "wms_title"             "mne250k blue"
        INCLUDE                 "../include/ec/ec_meta_layer.map"
    END
    PROJECTION
        "init=epsg:32198"
    END
    CLASS
        STYLE
            COLORRANGE 255 255 255   0 0 255
            DATARANGE -1 706
        END
    END
END
...
End the getMap query, use the group layer name "Ombrage_bleu_250K"... http://your.server.com/cgi-bin/mapserv?map=demshade&REQUEST=GetMap&SERVICE=WMS&VERSION=1.1.1&LAYERS=Ombrage_bleu_250K&STYLES=&FORMAT=image/png&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SRS=EPSG:32198&BBOX=87557.4046920096,498799.484254657,182346.544021857,583840.349445035&WIDTH=739&HEIGHT=663 In next example, I have tree layer that I can call with the GROUP tag "HILLSHAPE_GREEN_BLUE". First I add the hillshape in background. Then I overlay two MNE rasters layers that show elevation from -63 to 1280. So I coloring for the first -63 to 500 metre elevation with a green to blue colo rampe, and from 300 to 1280 with a blue to green color rampe. Not that all thos layer are add in the mapfile as Arc/INFO GRID TILEINDEX.

LAYER
    NAME  "Ombre_vert_bleu"
    GROUP  "HILLSHAPE_GREEN_BLUE"
    TYPE  RASTER
    TILEINDEX  "mneomb20k/omb"
    TILEITEM  "location"
    MAXSCALE  200000
    MINSCALE  1
    METADATA
        "wms_group_title"    "Ombrage  vert  bleu  20K"
        "wms_name"           "Ombre_vert_bleu"
        INCLUDE              "../include/ec/ec_meta_layer.map"
        "wms_title"          "Ombre"
    END
    PROJECTION
        "init=epsg:32198"
    END
END
  #-------------------------
LAYER
    NAME  "MNE_comp_green_blue"
    GROUP  "HILLSHAPE_GREEN_BLUE"
    TILEINDEX  "mneomb20k/mne"
    TILEITEM  "location"
    TRANSPARENCY  40
    TYPE  RASTER
    PROCESSING  "SCALE=-63,500"
    MAXSCALE  200000
    MINSCALE  1
    METADATA
        "wms_name"         "MNE_comp_green_blue"
        INCLUDE            "../include/ec/ec_meta_layer.map"
        "wms_title"        "MNE  comp  green"
    END
    PROJECTION
            "init=epsg:32198"
    END
    CLASS
        STYLE
            COLORRANGE  34  139  34  255  255  255
            DATARANGE  -63  500
        END
    END
END
#--
LAYER
    NAME  "MNE_comp_blue_green"
    GROUP  "HILLSHAPE_GREEN_BLUE"
    TILEINDEX  "mneomb20k/mne"
    TILEITEM  "location"
    TRANSPARENCY  40
    TYPE  RASTER
    PROCESSING  "SCALE=300,1280"
    MAXSCALE  200000
    MINSCALE  1
    METADATA
        "wms_name"      "MNE_comp_blue_green"
        INCLUDE         "../include/ec/ec_meta_layer.map"
        "wms_title"     "MNE  comp  bleu"
    END
    PROJECTION
        "init=epsg:32198"
    END
    CLASS
        STYLE
            COLORRANGE  255  255  255  0  0  205
            DATARANGE  300  1280
        END
    END
END

The result of this combination show here Hillshade green-blue

New Spatialite format with Mapserver and GDAL/OGR 1.7.0

Sometime we find something such simple as stupide ... I think that Spatialite is one of that.  If you look for a GIS storage format AND tabular data, very simple, open, easy to use and manage with Mapserver, python or php, take a look to this solution. Note that with a single database file you can build, query and manage your data like MySql or Postgresql. For my test, I use the open data Natural Earth, and it work at my first try ..! :) Rock On!! My NaturalEarth.sqlite file can be dowload here. (Natural Earth. Free vector and raster map data @ naturalearthdata.com). I load it with the Spatiallite-gui tools and than visualize them with the Spatiallite-gis... It take's me 15 minutes... Including the download and install steps... To make my first WMS Spatialite data test, I build a mapfile for Mapserver with this cool free vector dataset of Natural Earth. I just have to replace the shapefiles connection string with a OGR connection type and specify the table name:         CONNECTIONTYPE OGR         CONNECTION "path_to_spatialite_file"         DATA "50m-geography-marine-polys" Also, note that you HAVE to use the latest GDAL/OGR(version 1.7.0). If you use MS4W, you can download the beta10 or later. My first impression is that is not as fast as shapefile format. I gona test for larger dataset later... But, according to OGR specification web page, this driver still don't take advantage of spatial index?. So it's not a negative tips of this format... My mapfile test is here: MAP     NAME "spatialite"     EXTENT -180 -90 180 90       SIZE 700 500     IMAGETYPE PNG     IMAGECOLOR 0 0 0       UNITS METERS     SYMBOLSET                   "/ms4w/msp/symbols/commun/symbols.map"     FONTSET                     "/ms4w/msp/fonts/commun/fonts.txt"     CONFIG MS_ERRORFILE         "/ms4w/tmp/mapserv.log"     WEB         QUERYFORMAT     "text/xml"         BROWSEFORMAT    "text/xml"         IMAGEPATH       "/srv/www/msp/services/tmp/"         IMAGEURL        "/ms_tmp/"             METADATA            "wms_title"                  "spatialite test"            "wms_name"                   "spatialite"            "wms_abstract"               ""            "wms_description"            ""            "wms_keywordlist"            "Spatialite test"            "wms_onlineresource"         "http://localhost/cgi-bin/mapserv.exe?map=C:/Travail/spatialite/spatialite.map"         END     END     PROJECTION         "init=epsg:4326"     END     LAYER         NAME "50m-admin-1-states-provinces-shp"         DATA "C:/Travail/spatialite/NaturalEarth/50m-admin-1-states-provinces-shp.shp"          TYPE polygon         PROJECTION             "init=epsg:4326"         END         METADATA             "wms_name"              "50m-admin-1-states-provinces-shp"             "wms_title"             "states-provinces shapefiles"             "wms_keywordlist"       ""             "wms_server_version"    "1.1.1"         END         CLASS             NAME "states-provinces"             STYLE                 COLOR 255 90 90             END         END     END         LAYER         NAME "50m-admin-0-countries"         CONNECTIONTYPE OGR         CONNECTION "C:/Travail/spatialite/NaturalEarth/NaturalEarth.sqlite"  # full path to SQLite db file         DATA "50m-admin-0-countries"         TYPE polygon         PROJECTION             "init=epsg:4326"         END         METADATA             "wms_name"              "50m-admin-0-countries"             "wms_title"             "50m-admin-0-countries"             "wms_keywordlist"       ""             "wms_server_version"    "1.1.1"             "wms_extent"            "-180 -90 180 90"         END         CLASS             NAME "states-provinces"             STYLE                 COLOR 255 190 190             END         END     END      LAYER         NAME "50m-geography-marine-polys"         CONNECTIONTYPE OGR         CONNECTION "C:/Travail/spatialite/NaturalEarth/NaturalEarth.sqlite"  # full path to SQLite db file         DATA "50m-geography-marine-polys"          TYPE polygon         PROJECTION             "init=epsg:4326"         END         METADATA             "wms_name"              "50m-geography-marine-polys"             "wms_title"             "50m-geography-marine-polys"             "wms_keywordlist"       ""             "wms_server_version"    "1.1.1"             "wms_extent"            "-180 -90 180 90"         END         CLASS             NAME "states-provinces"             STYLE                 COLOR 24 116 205             END         END     END  END 

A new XML Mapfile Format in Mapserver 5.6

Build a mapfile for Mapserver from plain text is not an easy job.  We can use a Syntax coloring Editor like SciTE, PsPad or UltraEdit, but we all hope have a nice and easy  mapfile editor for build our web map service.  Sice version 5.6, Mapserver has a XML schema has been defined to encode mapfiles in XML format. We can get some input from Mapserver Wiki for example, how to implement it and how to convert existing mapfile. <?xml version="1.0" encoding="UTF-8"?> <Map name="GMAP-DEMO" version="5.6.0" status="ON" xmlns="http://www.mapserver.org/mapserver" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mapserver.org/mapserver"> <extent>-2200000 -712631 3072800 3840000</extent> <fontSet>../etc/fonts.txt</fontSet> <imageColor red="255" green="255" blue="255"/> <Layer name="bathymetry" type="RASTER" status="ON"> <data>bath_mapserver.tif</data> <Metadata> <item name="DESCRIPTION">Elevation/Bathymetry</item> </Metadata> </Layer> <Layer name="popplace" type="POINT" status="ON"> <Class name="Cities"> <color red="0" green="0" blue="0"/>< expression>1</expression> <Label type="TRUETYPE"> <align>LEFT</align> <color red="255" green="0" blue="0"/> <font>sans-italic</font> <outlineColor red="255" green="255" blue="255"/> <partials>FALSE</partials> <position>AUTO</position> <size>8</size> </Label> ...So this feature is not realy exciting BUT now we have a tool to parse any mapfile and option to build new client interface for Mapfile.  This is a prety good news... :-)

How to have nice symbol pixmap with transparency in Mapserver

First, to have a nice symbol pixmap, you should reduce the size of the image the same size it should appear in the map. To do this, you must not resize the symbol in the class layer of your mapfile.  It may be necessary to resample your image. Use Gimp(a photoshop Open sources) for this. For transparency works well, use a gif format. If your symbol(image) does not use transparency, you can quickly create one with Microsoft Photo Editor.  You only need to identify the background color with Transparent Color function.  This little function allows you to quickly create a NoData Value for image. Then, you must know the index(NoData Value) of transparent color of your symbole. This is not the transparent RGB color. To do this, you can use the utility gdalinfo to get the transparent index color. In this example, we have NoData Value = 254. Finally, just have to put this value in the symbol pixmap of your mapfile: SYMBOL    NAME "nn"    TYPE PIXMAP    IMAGE "1nn.gif"    TRANSPARENT 254 END

Optimizing your mapfile for Mapserver

There are many tricks to optimize your mapfile.  After many years to build mapfiles for Mapserver, I learned many trick to optimise a mapfile.  I try here to highlight some of them:
  1. Manage your EPSG file if you specify espg code.  Only keep in your file projections used in your systems.  The original file have approx 545k and Mapserver read this file each time you add in your mapfile instruction like  "init=epsg:32198".  You can reduce it to 4k. Or, you can put the projections specify in your mapfile at the top of your epsg file...  so Mapserver don't need to scan the whole epsg file.
  2. Lowercase epsg intruction like "init=epsg:32198".  Mapserver use C function "strcmp" in lowercase first.
  3. Try to be "brief"... Don't put extra default instruction like STATUS ON or LABELCACHE ON... STATUS and LABELCACHE are ON by default.  The less you have to read, faster you are. And try to limit your comment to...
  4. In the layer definition the tag EXPRESSION can be very costly.  Alway try to use "regular expression"(REGEX) it's much faster!  If you add something like that EXPRESSION (("[DESCRIPTION]" eq "Park") OR ("[DESCRIPTION]" eq "Base") Mapserver gona built sql string then gona filtre your data! It work but it's not optimize.  In this example we just put directly in the mapfile EXPRESSION /^Park$|^Base$/ and Mapserver use it without any string manipulation.  And change this EXPRESSION ("[DESCRIPTION]" eq "Park") to this EXPRESSION "Park"!
  5. Try not to re-project your data it's costly overhead.
  6. Shapefile format ALWAYS de faster way to publish data.
  7. Shptree each of your shapefile AND don't specify shp in DATA tag.  This one is very peculiar one! If you put DATA like this DATA "/srv/data/park.shp" Mapserver use a built in reader and don't use index qix file.  But if you type DATA "/srv/data/park" Mapserver gona use OGR to read data and OGR alway use qix file!
  8. For large shapefiles, use tile4ms program to tile them.  Follow those instruction to split your shapefile.  This trick is very very usefull
  9. If you use PostGIS or Oracle Data, try to connect to only one server and keep your connection open with this instruction in each layer PROCESSING "CLOSE_CONNECTION=DEFER"
  10. Never use SIZEUNITS in the LAYER tag.  Put it in the MAP tag.
  11. If you build a mapfile for a Web map service(WMS) always add STATUS OFF in each layer.  In this way Mapserver don't prepared all cartographics parametres like labeling engine.
  12. Finally, use debug option to find out witch layers are to slow.  The best trick her is to use shp2img program to eliminate overhead of your network.  To use debug option you have to insert in the MAP tag this CONFIG MS_ERRORFILE "/srv/log/debug.log" and in each layer this tag DEBUG 5  aIn the log file check the msDrawMap for each layer.
Shp2imp example on Windows shp2img -m C:Testimg2.map -l png -o C:Testext.png -e 219589 5347860 226045 5353101 Log example [Fri Feb 15 16:05:14 2008].742000 msDrawRasterLayerLow(tif): entering. [Fri Feb 15 16:05:14 2008].762000 msDrawGDAL(): src=0,0,33477,25307, dst=0,26,935,707 [Fri Feb 15 16:05:14 2008].762000 msDrawGDAL(): red,green,blue,alpha bands = 1,2,3,0 [Fri Feb 15 16:05:15 2008].2000 msDrawMap(): Layer 2 (tif), 0.260sp