Importing Extracted Data taking too long

[StarGPS]

Hello everybody,
This is my first post in this honorable community.
I'm trying to have my own server for reverse-geocoding purposes on:
4 cores * 4 threads CPU, 16GB RAM, and 500GB HDD VM.
I was able to import the African extracted data relatively pretty quickly, but when I try to import the EU and African data together it is taking too long. It is now more than 48H and the import process is still unfinished. Following is what I get when I run the import command:

nominatim@nominatim:~/nominatim-project$ NOMINATIM_IMPORT_STYLE=street nominatim import --osm-file africa-latest.osm.pbf --osm-file europe-latest.osm.pbf --reverse-only --no-updates 2>&1 | tee setup.log
2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(152112k 0.51k/s)** Relation(0 0.0/s)

I can eventually wait, but I'm wondering if this is a normal speed for the "Way" part. As I tried before to import the EU+NA+AF and it ended with an error of the import process being aborted abnormally.

Thanks in advance for your help!

[mtmail]

For Europe you should use a flatnode file https://nominatim.org/release-docs/latest/admin/Import/#flatnode-files And of course faster hardware, more RAM, faster local harddrives always help.

[StarGPS]

Hi, Thanks for the quick reply! I'm indeed using a flatnode file to import data.

…

On Mon, Aug 21, 2023 at 1:29 PM mtmail ***@***.***> wrote: For Europe you should use a flatnode file https://nominatim.org/release-docs/latest/admin/Import/#flatnode-files And of course faster hardware, more RAM, faster local harddrives always help. — Reply to this email directly, view it on GitHub <#3160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKAY3Z5GURQD7HLEO6ORYA3XWPALRANCNFSM6AAAAAA3Y2Y3O4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[StarGPS]

So this is a normal speed for my hardware config?

[StarGPS]

2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(415901k 0.87k/s)** Relation(0 0.0/s)

+5 days now and it is still going!

[StarGPS]

Just an update to whom might be interested. Finally the Europe and Africa Extracted data has been imported successfully. Here is the timeline:

2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(415901k 0.87k/s)** Relation(0 0.0/s)

Reading input files done in 587587s (163h 13m 7s).
2023-08-25 02:52:04    Processed 4327447553 nodes in 3063s (51m 3s) - 1413k/s
2023-08-25 02:52:04    Processed 522215618 ways in 532631s (147h 57m 11s) - 980/s
2023-08-25 02:52:04    Processed 7333836 relations in 51893s (14h 24m 53s) - 141/s

`2023-08-30 03:26:42: Setup website at /srv/nominatim/nominatim-project/website`

It took roughly 11 days and 20 hours to complete the process.

Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  489G   75G  393G  17% /

Although the import process in my case needed about 300GB disk space to process the import. In the end, most of the data has been dropped and only 50GB of storage is being used if you use the --reverse-only --no-updates parameters.

Patience and luck is needed here :)

[lonvia]

Thanks for the report on the outcome.

Your hardware was on the very low end. For an import that involves Europe, I couldn't recommend anything with less than 32GB of RAM, 8 CPUs and fast SSDs. But good to see that with patience less hardware will still do.