Remote
OpenStreetMap Ops Team
@osm_tech@en.osm.town
mastodon 4.7.33287 Followers
20 Following
50 Posts
Joined July 10, 2023
openstreetmap-website, the codebase that powers our website and API, turns 20 years old today!
A big thank you to all the awesome people who built the code that powers #OpenStreetMaphttps://community.openstreetmap.org/t/20-years-of-openstreetmap-website/145877 🎂 ❤️ 
https://OpenStreetMap.org has been disrupted today. We're working to keep the site online while facing extreme load from anonymous scrapers spread across 100,000+ IP addresses. Please be patient while we mitigate and protect the service. #OpenStreetMap #DDoS #Scrapers #AI
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
🙏🌍 #AI #Bots #Abuse
To help handle #QGIS load on tile.openstreetmap.org, a new dedicated server, quetzal, joins the QGIS pool today. 🚀
Thank you to @qgis@fosstodon.org for supporting the hardware purchase!
https://hardware.openstreetmap.org/servers/quetzal.openstreetmap.org/
Do you know anyone at Upwork who could help us reach the right team?
OpenStreetMap is seeing repeated unauthorised spam from Upwork freelancers. It wastes our time and damages the reputation of the businesses using these "SEO" services.
We have switched to using https://pgBackRest.org for backups of our primary database 🗄️, covering both #PostgreSQL base backups and WAL archives. This has allowed us to optimise the weekly planetfile-generation pipeline 🌍, these weekly exports will now be published earlier in the week. A big thank you to the #pgBackRest team for a great product! 🙌
We're simplifying privacy options for new GPS trace uploads, making it clearer what permission you are granting. This change will also enable future performance improvements to the GPS points API, which is currently slowed by supporting several different privacy models.
https://community.openstreetmap.org/t/changes-coming-to-gps-trace-uploads/145728
The maintenance is now complete and the database is back online
OpenStreetMap.org's Standard Map Layer is now running the latest release of OSM Carto: v6.0.0 🗺️ 🎉
This is a major release with a significant number of improvements. Huge thanks to everyone involved in making it happen 🙌
More details:
https://www.openstreetmap.org/user/imagico/diary/408344
https://github.com/openstreetmap/chef/issues/841
https://github.com/openstreetmap-carto/openstreetmap-carto/issues/5184
We are aware of the issues with the tile.openstreetmap.org service at the moment and are investigating. #OpenStreetMap #DDOS
UPDATE: Service is now returning to normal, but we are continuing to investigate cause.
Replying to
Maintenance has been completed and the service is back to normal.
Replying to
@ISibboI Correct. Not an ounce of brain. It has been going on for months, but it is just getting worse. Waste of their resources/time and ours.
The app was responsible for over 2k req/s. Here is when we blocked it. Traffic is a portion of requests we serve in India.
There will be a brief period of downtime on the main database at 08:00 UTC to allow some maintenance to be carried out - the downtime should only last five minutes or so
UPDATE: The maintenance is now complete and the database is back online.
No #OpenStreetMap planet file has been published for this week, next release will be back to normal schedule. Please use diffs if you cannot wait until then.
We had a DB permission mistake which was caught too late to fix this week's planet.
Replying to
@geospacedman @ISibboI They are scraping the website pages: /ways/, /nodes/ and /relations/. All this data is published on planet.osm.org already.
Replying to
Fastly's VCL + bot management is incredibly powerful. Running custom logic directly at the edge gives us huge flexibility when protecting and operating #OpenStreetMap services. ⚙️
Replying to
#OpenStreetMap community forum https://community.openstreetmap.org/ is back up and running. Maintenance completed successfully.
Replying to
#QGIS users? Ensure you have #OpenStreetMap tile resolution set correctly to "Standard" resolution for improved service.
We've migrated our server configuration management to CINC - the free-as-in-freedom build of Progress Chef.
Huge thanks to the CINC team ❤️
https://cinc.sh/
CC: @ramereth@fosstodon.org
#OpenSource #DevOps #Infrastructure #SRE
We've temporarily removed the block. We're still waiting to hear from Delhivery: Our contact details are available here: https://operations.osmfoundation.org/
Sorry to all their customers with delayed deliveries. How about mapping your area while you wait for your parcel? 😉 https://www.openstreetmap.org/fixthemap 🗺️ ❤️
Replying to
@Don_Clemente Just dumb scrapers. They want our data, so maybe we /should/ redirect them to the latest full history planet file, it is only 245GB ;-)
Replying to
@boz We will survive and grow stronger thanks to our awesome mapping community :-) We continue to squeeze more capacity out of our servers, but eventually we'll need to upgrade despite the $$$ RAM prices.
Replying to
@BalooUriza We use fail2ban to handle some of this with custom rules, but eventually fail2ban becomes a bottleneck after 100,000 IP addresses.
Replying to
@atvirasis@mapstodon.space Yes, we have raised this with the QGIS team, there have been improvements in very recent releases of QGIS. Best to upgrade to the latest version if possible. Some more info here: https://community.openstreetmap.org/t/severe-disruption-to-qgis-openstreetmap-tile-openstreetmap-org-users/143779
Replying to
@utf_7 It is madness, start here: https://www.openstreetmap.org/node/1 and keep going once you reach https://www.openstreetmap.org/node/10000000000 then start on ways, and relations :-P or just download the latest weekly export from planet.openstreetmap.org 😏
Replying to
@luis_in_brief Historically we've been able to run OpenStreetMap very lean: a small volunteer sysadmin/SRE team and minimal, low-cost refurbished hardware. Both are now being pushed to breaking point. And most annoyingly, the scrapes are trying to get data we ALREADY PUBLISH as complete weekly compressed downloads on planet.osm.org
Replying to
@michel42 We'd like to share the IP address list, but unfortunately don't think we can due to legal concerns.
Replying to
@gme Anubis is great, but not ideal for us. We have reasonable technical mitigations in place, but it is unfortunately an arms race (and time/resource blackhole)
Replying to
Replying to
@JonSaenzAgirre It is a good questions, and we don't know the answer either. Our planet data is so much easier to process and use.
Replying to
@utf_7 We've had 400,000 IPs in the last 24 hours. Each IP only does a few requests. Technically we're managing, but no fun fighting this daily rather than building new things.
Replying to
@LMieldazis @geerlingguy oooh do we get to show him our out-of-band (remote access) Raspberry Pi with dual power feeds, 4G modem and loads of serial connections? Saved our skin a good few times.
Replying to
@thibaultmol Small part, but the log aggregation and analysis is all done within the EU.
Replying to
@fm_volker We do, here is an example: https://pnorman.github.io/osm-leaflet-demo/ (error they receive is slightly different)
Replying to
@thibaultmol@en.osm.town @th@social.v.st Thanks for the nudge, we are in bad need of extra tile render capacity, but we likely need newer kit. NVMe + additional power efficient cores. Our rack in Amsterdam is very power limited.
Replying to
2025: Servers on Fire: Keeping OpenStreetMap Online
Replying to
@mdione@en.osm.town @Cyrille37@mamot.fr Most of our servers now run Debian 12 or 13. Yes, our DB servers still run Ubuntu LTS using official upstream postgres packages, we haven't yet upgraded OS because of the complexity and size of what needs upgrading. DB collation is affected by the underlying OS.
Replying to
@eigen@mattstodon.panar.ooo we're always looking for more volunteers to join our team. ♥️
Replying to
@mdione Per delhivery.com website: "We are India's largest fully integrated logistics services provider"