If you ever watched 'Person of Interest' you'll be aware of something called 'The Machine'. Basically, a massive computer that watches over the general public and helps to predict crime/terrorism before it occurs (There are organisations that have been pursuing such (on a more limited scale) research/work around the world.).
I've been thinking about how to build such a machine (I may try for a very limited prototype if I have time.). When you think about it all it's really doing is pushing the concept of data mining (search engine technology is common now and mostly static with regards to what it does with the data) to its most extreme. Let's say you have control over all the surveillance feeds out there (Forget issues of jurisdiction and so on. This is pure theory. Software that allow for automated 'penetration testing' is now available for research/commercial purposes if we don't have access to a clean/unprotected feed.), then it would be a case of being able to convert visual, audio, textual and other information into a form that can be analysed by existing computer systems (practical solutions for Optical Character Recognition (OCR), speech recognition, and so on have come a long way. A decade ago I recall experimenting/working with a University research (Sphinx) on a P3 laptop. Even after reducing the volcabulary and using a preamble processing time was comical. Moreover, even commercial solutions had their problems with real time, proper speech recognition. Now, you can get reasonably accurate voice recognition for under 200, OCR technology is widely used and deployed commercially, and data mining technologies are used everywhere from your Operating System to search engines such as Google, Yahoo, and Bing.).
Thereafter, it's a case of developing my 'Automated Research/Reporting/Analyses' concepts further. The more practical problems revolve around storage capacity, computing power, and so forth (though I suspect that Quantum computing has made significant strides behind closed doors). One big problem is the problem of 'context' though. It doesn't matter if your machine has all the power/capacity in the world. If it doesn't apply the correct algorithm to the problem the solution will never be correct.
Ironically some information (nothing critical, illegal. Just some technical information.) which was sought during my research has been easier to find in countries which have been supposedly been under some form of sanction/s. The other thing that also seems clear that while the 'bad guys' are fairly open with tools and information the same can not necessarily be said of the 'good guys'. While there are various programs to address this (The US DHS have been pursuing the notion of a 'CyberCorps', research material in some Universities is fairly openly available, and there are some commercial entities that are providing more than just advertising/marketing material when it comes to actual security analysis.) I believe that there is still a massive gap between where the 'good guys' and the 'bad guys' are. Education programs are 'gappy' and pursuing a career in this particular sector of the industry itself is hard even if you have a solid technology background.
In spite of extensive firewalling and filtering there seems to be wide spread knowledge of how to get around these particular problems in highly policed states.
Many people are concerned with so called, 'Rules of Engagement' in cyberspace at the moment. If history is anything to go by, if you open up you will generally develop a set of rules/morals/ethics which most people agree by in terms of war. Obviously, some people/states are willing to push it further than others though. What's acceptable in the East, West, and Middle East, are often very much different. Moreover, in intelligence/war things aren't always clear cut and you need to adapt to your circumstances.
Have been thinking further about concept of 'Algorithmic Masking'. Need to be careful with regards to being able to infer classified data from un/declassified data.
May have found a bug in blogger.com Under certain conditions/circumstances it will refuse to publish and a warning will be provided. Results in post being posted repeatedly even though it states that a bug has been found. Believe that it may be browser related but am working on other stuff at the moment. May test later on.