20 Oct
2022
20 Oct
'22
9:01 a.m.
Hi,
If you configure it (remarks and/or geofeeds: attribute in whois), they will come.
I see a bit below a thousand IPs fetching my geofeed data over the last few days
Quick grep (with some filtering due to versions etc) of UAs; see below; quite some normal
web crawlers, but also dedicated it seems.
Note that like anything, geofeed is just an indicator, somebody might use or not use it.
Many build precise geoloc based of physical orders and/or GPS in mobile phones, thus
depending on users logging in to certain websites or ordering things and having GPS
enabled for that site...
Greets,
Jeroen
--
"APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)"
"CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)"
"Crawlbot/Nutch-1.19-SNAPSHOT"
"Go-http-client/2.0"
"IonCrawl (https://www.ionos.de/terms-gtc/faq-crawler-en/)"
"Ipregistry/2.0.0 (geofeeds; +https://ipregistry.co)"
"Mozilla/5.0 (Windows NT 6.1; Win64; x64;
+http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0"
"Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)"
"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
"Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)"
"Mozilla/5.0 (compatible; Crawlson/1.0;
+https://www.crawlson.com/search?q=site:as57777.net)"
"Mozilla/5.0 (compatible; DataForSeoBot/1.0;
+https://dataforseo.com/dataforseo-bot)"
"Mozilla/5.0 (compatible; Dataprovider.com)"
"Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot,
help(a)moz.com)"
"Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot;
help(a)moz.com)"
"Mozilla/5.0 (compatible; EntferBot/0.1; +https://entfer.com;
+a_new_independent_search_engine;)"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
"Mozilla/5.0 (compatible; Linespider/1.1; +https://lin.ee/4dwXkTH)"
"Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"
"Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"
"Mozilla/5.0 (compatible; SeznamBot/4.0-RC1;
+http://napoveda.seznam.cz/seznambot-intro/)"
"Mozilla/5.0 (compatible; TorusBot/1.0; +https://torus.company/bot.html)"
"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
"Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)"
"Mozilla/5.0 (compatible; ips-agent)"
"Python/3.9 aiohttp/3.8.1"
"Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)"
"Wget/<various>"
"axios/0.27.2"
"clark-crawler2/Nutch-1.19-SNAPSHOT"
"colly - https://github.com/gocolly/colly/v2"
"curl/<various>"
"db-ip
"libwww-perl/6.42"
"ltx71 - (http://ltx71.com/)"
"netEstate NE Crawler (+http://www.website-datenbank.de/)"
"python-requests/2.22.0"
"serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/;
abuse(a)serpstatbot.com)"
"x28-job-bot; +http://x28.ch/bot.html"
On 17 Oct 2022, at 11:39, Fredy Kuenzler
<kuenzler(a)init7.net> wrote:
I think that the RFC8805 approach is the way to go for the industry, however I wonder
which ISP is already using it in real life today. I couldn‘t find much about it besides
the slides of RIPE81.
We see GeoIP complaints of end customers on a regular basis and it would be nice if
RFC8805 could solve the problem.
https://ripe81.ripe.net/presentations/27-RIPE81_geofeeds_discovery.pdf
27-RIPE81_geofeeds_discovery
PDF-Dokument · 2.3 MB
--
Fredy Künzler
Init7 (Switzerland) Ltd.
Technoparkstrasse 5
CH-8406 Winterthur
https://www.init7.net/
<(null).(null)>_______________________________________________
swinog mailing list -- swinog(a)lists.swinog.ch
To unsubscribe send an email to swinog-leave(a)lists.swinog.ch
20 Oct
20 Oct
5:21 p.m.
New subject: MaxMind location determination (was: Contact: geoiplookup.net)
My small experience in AS210312 is the following:
1. MaxMind
I have an alert to find incorrect geolocation (so far just country-level), and then submit
this: https://www.maxmind.com/en/geoip-location-correction
<https://www.maxmind.com/en/geoip-location-correction>
Usually within 1-2 weeks the alert goes away. It fires mainly for networks with VPNs.
I had a /22 geolocated in CH, and I was getting frequent alerts, but then I split that and
submitted information for /27-/32 (IPv4) to MaxMind authoritatively, proactively, for
almost all addresses (did this once) and then all problems are much less frequent. I think
more specifics here beat their automated /25 solution. Perhaps it would work if you submit
all your addresses as /26 :)
I tried to get them to accept my GeoFeed, as per
https://www.maxmind.com/en/geoip-data-correction-request#batch-corrections
<https://www.maxmind.com/en/geoip-data-correction-request#batch-corrections> but
they told me they won’t, and they don’t respect any geofeeds in RIPE, etc. and it’s only
the ones they get in their program.
Perhaps a large ISP can be more convincing.
2. All other IP geolocation services
They seem to generally follow MaxMind because I did nothing, and they picked up
corrections after a few weeks. Or they just fight much harder to be more accurate, to gain
market share. I never submitted correction reports. Most are actually respecting GeoFeeds,
so when I added it to the RIPE inet{6,}num objects (even with more specifics), they
started fetching it frequently, and within hours of updating the RIPE DB. From the User
Agents, I see at least:
* db-ip
* Ipregistry
* Someone using axios
* Someone using Go
* Someone using Python
My list of agents is much smaller than Jeroen’s but with similar entries.
3. Google
The only way I found was getting access to the Peering Portal and adding the feed there
manually. Then, it’s a few days and things seem fixed, and not changing randomly a lot.
- - -
I have not confirmed this, but I have a suspicion that at least one of the “others”
invalidates the entire file if there’s a bad line. For anycast I used the country code
“UN” (to signify whole world) which is the best fit according to the ISO referenced in the
RFC. Apparently this is invalid for someone, and they didn’t process the file. I switched
this to “EU” and this was fine. Google, in their ISP Peering Portal, will complain that
“EU” is invalid, but they just ignore that line, not the entire file.
Perhaps we need a study into GeoFeeds and how they are processed / interpreted, as it
seems that it’s up to interpretation by some..
Antonis
On 20 Oct 2022, at 09:01, Jeroen Massar
<jeroen(a)massar.ch> wrote:
Hi,
If you configure it (remarks and/or geofeeds: attribute in whois), they will come.
I see a bit below a thousand IPs fetching my geofeed data over the last few days
Quick grep (with some filtering due to versions etc) of UAs; see below; quite some normal
web crawlers, but also dedicated it seems.
Note that like anything, geofeed is just an indicator, somebody might use or not use it.
Many build precise geoloc based of physical orders and/or GPS in mobile phones, thus
depending on users logging in to certain websites or ordering things and having GPS
enabled for that site...
Greets,
Jeroen
--
"APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html
<https://developers.google.com/webmasters/APIs-Google.html>)"
"CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html
<http://www.checkmarknetwork.com/spider.html>)"
"Crawlbot/Nutch-1.19-SNAPSHOT"
"Go-http-client/2.0"
"IonCrawl (https://www.ionos.de/terms-gtc/faq-crawler-en/
<https://www.ionos.de/terms-gtc/faq-crawler-en/>)"
"Ipregistry/2.0.0 (geofeeds; +https://ipregistry.co
<https://ipregistry.co/>)"
"Mozilla/5.0 (Windows NT 6.1; Win64; x64;
+http://www.komodia.com/newwiki/index.php/URL_server_crawler
<http://www.komodia.com/newwiki/index.php/URL_server_crawler>) KomodiaBot/1.0"
"Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/
<https://seostar.co/robot/>)"
"Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/
<http://ahrefs.com/robot/>)"
"Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler
<https://babbar.tech/crawler>)"
"Mozilla/5.0 (compatible; Crawlson/1.0;
+https://www.crawlson.com/search?q=site:as57777.net
<https://www.crawlson.com/search?q=site:as57777.net>)"
"Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot
<https://dataforseo.com/dataforseo-bot>)"
"Mozilla/5.0 (compatible; Dataprovider.com <http://dataprovider.com/>)"
"Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot
<http://www.opensiteexplorer.org/dotbot>, help(a)moz.com
<mailto:help@moz.com>)"
"Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot
<https://opensiteexplorer.org/dotbot>; help(a)moz.com
<mailto:help@moz.com>)"
"Mozilla/5.0 (compatible; EntferBot/0.1; +https://entfer.com
<https://entfer.com/>; +a_new_independent_search_engine;)"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
<http://www.google.com/bot.html>)"
"Mozilla/5.0 (compatible; Linespider/1.1; +https://lin.ee/4dwXkTH
<https://lin.ee/4dwXkTH>)"
"Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/
<http://mj12bot.com/>)"
"Mozilla/5.0 (compatible; MegaIndex.ru/2.0 <http://megaindex.ru/2.0>;
+http://megaindex.com/crawler <http://megaindex.com/crawler>)"
"Mozilla/5.0 (compatible; SeznamBot/4.0-RC1;
+http://napoveda.seznam.cz/seznambot-intro/
<http://napoveda.seznam.cz/seznambot-intro/>)"
"Mozilla/5.0 (compatible; TorusBot/1.0; +https://torus.company/bot.html
<https://torus.company/bot.html>)"
"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots
<http://yandex.com/bots>)"
"Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd
<http://naver.me/spd>)"
"Mozilla/5.0 (compatible; ips-agent)"
"Python/3.9 aiohttp/3.8.1"
"Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots
<https://api.slack.com/robots>)"
"Wget/<various>"
"axios/0.27.2"
"clark-crawler2/Nutch-1.19-SNAPSHOT"
"colly - https://github.com/gocolly/colly/v2
<https://github.com/gocolly/colly/v2>"
"curl/<various>"
"db-ip
"libwww-perl/6.42"
"ltx71 - (http://ltx71.com/ <http://ltx71.com/>)"
"netEstate NE Crawler (+http://www.website-datenbank.de/
<http://www.website-datenbank.de/>)"
"python-requests/2.22.0"
"serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/
<https://serpstatbot.com/>; abuse(a)serpstatbot.com
<mailto:abuse@serpstatbot.com>)"
"x28-job-bot; +http://x28.ch/bot.html <http://x28.ch/bot.html>"
223
days inactive
223
days old
1 comments
2 participants
participants (2)
-
Antonios Chariton (daknob) -
Jeroen Massar