I’ve been thinking about something along these lines for some time. I really like the direction of this.
The challenge I see more broadly is we (as engineers now empowered by LLMs) are trying to find the right level of abstraction to operate in. Writing long form sentences and (sometime) reviewing the output feels too far away. But having an LLM work directly with you in an IDE feels too close to “the old way”.
Personally for me the approach here still feels a little too close to the lower level old way, but it’s better than the two approaches above.
Right level of abstraction is a good way of putting it. It's basically like creating a custom DSL, but flexibility of LLMs allow the DSL to be ad-hoc.
At what point will you need formal rigid syntax? Or is not having rigid syntax the point? If the latter, how much "informational noise" or ambiguity can you inject before the "DSL compiler" gets confused?
Scaling is another bit. Convertible Psuedocode a great pattern for writing functions, but is it useful for writing modules? If you're writing a paragraph to change behavior of a function, you're underutilizing LLMs. Paragraphs are best for spec'ing modules, and the LLMs already fill in the blanks. Not sure if it would be faster to psuedocode the entire module (although maybe just the interface would be a sweet spot...)
Yeah exactly. The module/directory level is currently untested. I'm working on a desktop version so I can talk to a file system, and then I'll be able to explore those problems.
My guess is that if you simply write `use some_fn from $repo/some/path`, the LLM _should_ be smart enough to infer in most cases. But we'll have to see how reliable that is.
I am new to OSINT and this looks like a really cool way to get started.
Maybe this exists and I missed it but it would nice to have a way not to run scans with a missing API key. Even better would be a way to easily link to instructions to obtain missing API keys, for example each module could have a metadata slot for 'API key generation url'... it would be a lot of work so I understand if it's not in the roadmap.
Company data is a little there, using OpenCorporates’ API. The tool originated with a smaller scope initially and has grown over 8 years of development. I can imagine how different it will look in another few years.
Actually you can target a bunch of things beyond domains, including IPs, usernames, phone numbers and more. And historic Whois and leak database modules are indeed there. In some cases you need API keys though but most offer free tiers for low volumes.
But yes, it’s not covering some of the other sources you mentioned... yet.
Yeah, this looks pretty awesome! I ran it against one of my own domains and there's a fair bit of false positive/flood of information, but the data is really interesting!
I didn’t even consider the MSF compatibility need for people, so will take that into consideration for a future release. The sfcli.py CLI was a starting point for that kind of functionality but not MSF compatible. Thanks for the feedback though!
Author of the project (not the OP) here. Must say it was quite a surprise to see this land in my HN feed today! I’ll do my best to answer the points raised below but if you have any questions or further feedback, I’m glad to hear it!
The challenge I see more broadly is we (as engineers now empowered by LLMs) are trying to find the right level of abstraction to operate in. Writing long form sentences and (sometime) reviewing the output feels too far away. But having an LLM work directly with you in an IDE feels too close to “the old way”.
Personally for me the approach here still feels a little too close to the lower level old way, but it’s better than the two approaches above.
Excited to see where you take it!