On 'Simulated Worlds'

OpenAI’s video generation model Sora, in its current iteration, is incredible even though it’ll undoubtedly get better in the coming months. The post they have on their website makes for fascinating reading.

Extending generated videos. Sora is also capable of extending videos, either forward or backward in time.

Long-range coherence and object permanence. A significant challenge for video generation systems has been maintaining temporal consistency when sampling long videos. We find that Sora is often, though not always, able to effectively model both short- and long-range dependencies. For example, our model can persist people, animals and objects even when they are occluded or leave the frame. Likewise, it can generate multiple shots of the same character in a single sample, maintaining their appearance throughout the video.

Interacting with the world. Sora can sometimes simulate actions that affect the state of the world in simple ways. For example, a painter can leave new strokes along a canvas that persist over time, or a man can eat a burger and leave bite marks.

These advancements, alongside how far LLMs and other transformer-based technologies have come in the past few months, have been quite something to behold. While equal parts exciting and terrifying, it’s hard not to think about how and how much they’ll impact industries and society at large. It will likely become harder (and more time-consuming) to sift through what is a genuine advancement and not just another grift (NFTs, anyone?). Art, music, technology, video games, programming, editing, writing, law, disinformation, misinformation, capital markets, cybersecurity, democracy, and medicine will all invariably see some impact. A small part of me thinks that, as amazing as all this is right now, ‘AI’ (quotes intentional) may not be immune to enshittification—not just from the pressure to monetise but also from the unstoppable deluge of low-quality and unimaginative generated content.

On Worldcoin, DAOs and digital identity black markets

Molly White has a great essay on Sam Altman’s iris scanning orb and its purported use cases.

Much of Worldcoin’s promises are predicated on the questionable idea that highly sophisticated artificial intelligence, even artificial general intelligence, is right around the corner. It also hinges on the “robots will take our jobs!” panic — a staple of the last couple centuries — finally coming to bear. Worldcoin offers other use cases for its product too, like DAO voting, but it is not the promise to solve DAO voting that earned them a multi-billion dollar valuation from venture capitalists.

Other use cases that Worldcoin has offered seem to assume that various entities — governments, software companies, etc. — would actually want to use the Worldcoin system. This seems highly dubious to me, particularly given that many governments have established identification systems that already enjoy widespread use. Some even employ biometrics of their own, like India’s Aadhaar. There’s also the scalability question: Worldcoin operates on the Optimism Ethereum layer-2 blockchain, a much speedier alternative to the layer-1 Ethereum chain to be sure, but any blockchain is liable to be a poor candidate for handling the kind of volume demanded by a multi-billion user system processing everyday transactions.

…

What will happen when you promise people anywhere from $10 to $100 for scanning their eyeball? What if that’s not dollars, but denominated in a crypto token, making it appealing to speculators? And what if some people don’t have the option to scan their own eyeballs to achieve access to it?

A black market for Worldcoin accounts has already emerged in Cambodia, Nigeria, and elsewhere, where people are being paid to sign up for a World ID and then transfer ownership to buyers elsewhere — many of whom are in China, where Worldcoin is restricted. There is no ongoing verification process to ensure that a World ID continues to belong to the person who signed up for it, and no way for the eyeball-haver to recover an account that is under another person’s control. Worldcoin acknowledges that they have no clue how to resolve the issue: “Innovative ideas in mechanism design and the attribution of social relationships will be necessary.“ The lack of ongoing verification also means that there is no mechanism by which people can be removed from the program once they pass away, but perhaps Worldcoin will add survivors’ benefits to its list of use cases and call that a feature.

Relatively speaking, scanning your iris and selling the account is fairly benign. But depending on the popularity of Worldcoin, the eventual price of WLD, and the types of things a World ID can be used to accomplish, the incentives to gain access to others’ accounts could become severe. Coercion at the individual or state level is absolutely within the realm of possibility, and could become dangerous.

On Facial Recognition and Identity Proofing

Wired has a good piece on the IRS in the US caving to public outcry and ditching its integration with ID.me - a service that was supposed to verify identities (by matching video selfies to existing records). It’s understandable why this would cause concerns given that facial recognition is rife with false matches, biases and a reputation for invasiveness. With fraud being a pressing issue now when a majority of us (at least in Australia) access nearly every civic service online, governments are going to want to think about how they balance policy, privacy and messaging.

Unfortunately, the landscape at the moment is messy and populated by a number of third-party vendors still finding their feet in an area where privacy and policy concerns are outweighed by sexier usability and convenience use cases.

“The fact we don’t have good digital identity systems can’t become a rationale for rushing to create systems with Kafkaesque fairness and equity problems.” - Jay Stanley, ACLU

It’ll be interesting to see how Australia’s Trusted Digital Identity Framework (TDIF) will look to address some of these inherent problems through a continuous expansion of its standards.

On web3

(Note to self)

Moxie has some great thoughts on web3 even though what it really means depends on who you ask.

NFTs:

Instead of storing the data on-chain, NFTs instead contain a URL that points to the data. What surprised me about the standards was that there’s no hash commitment for the data located at the URL. Looking at many of the NFTs on popular marketplaces being sold for tens, hundreds, or millions of dollars, that URL often just points to some VPS running Apache somewhere. Anyone with access to that machine, anyone who buys that domain name in the future, or anyone who compromises that machine can change the image, title, description, etc for the NFT to whatever they’d like at any time (regardless of whether or not they “own” the token). There’s nothing in the NFT spec that tells you what the image “should” be, or even allows you to confirm whether something is the “correct” image.

web3 and Platforms (web2):

The people at the end of the line who are flipping NFTs do not fundamentally care about distributed trust models or payment mechanics, but they care about where the money is. So the money draws people into OpenSea, they improve the experience by building a platform that iterates on the underlying web3 protocols in web2 space, they eventually offer the ability to “mint” NFTs through OpenSea itself instead of through your own smart contract, and eventually this all opens the door for Coinbase to offer access to the validated NFT market with their own platform via your debit card.

At the end of the stack, NFT artists are excited about this kind of progression because it means more speculation/investment in their art, but it also seems like if the point of web3 is to avoid the trappings of web2, we should be concerned that this is already the natural tendency for these new protocols that are supposed to offer a different future.

Clients and Servers in the Decentralised Web:

One thing that has always felt strange to me about the cryptocurrency world is the lack of attention to the client/server interface. When people talk about blockchains, they talk about distributed trust, leaderless consensus, and all the mechanics of how that works, but often gloss over the reality that clients ultimately can’t participate in those mechanics. All the network diagrams are of servers, the trust model is between servers, everything is about servers. Blockchains are designed to be a network of peers, but not designed such that it’s really possible for your mobile device or your browser to be one of those peers.

With the shift to mobile, we now live firmly in a world of clients and servers – with the former completely unable to act as the latter – and those questions seem more important to me than ever. Meanwhile, ethereum actually refers to servers as “clients,” so there’s not even a word for an actual untrusted client/server interface that will have to exist somewhere, and no acknowledgement that if successful there will ultimately be billions (!) more clients than servers.

The increasing complexity of creating software:

At this point, software projects require an enormous amount of human effort. Even relatively simple apps require a group of people to sit in front of a computer for eight hours a day, every day, forever. This wasn’t always the case, and there was a time when 50 people working on a software project wasn’t considered a “small team.” As long as software requires such concerted energy and so much highly specialized human focus, I think it will have the tendency to serve the interests of the people sitting in that room every day rather than what we may consider our broader goals. I think changing our relationship to technology will probably require making software easier to create, but in my lifetime I’ve seen the opposite come to pass. Unfortunately, I think distributed systems have a tendency to exacerbate this trend by making things more complicated and more difficult, not less complicated and less difficult.

On the SolarWinds Breach

Where to begin. It’s almost impossible to comprehend what the fallout of this breach will be in the immediate to medium term; in fact, there isn’t enough information out there yet to conduct an effective post-mortem so to speak.

One thing is for certain - organisations are going to be wary of trusting ‘technology solutions’ from vendors. This isn’t to say that SolarWinds (and FireEye) did not have adequate measures in place; just that breaches are inevitable and organisations that rely on technology vendors are also dependent on these vendors having adequate controls in place alongside stringent self-audits. Organisations out there that trusted SolarWinds to push Orion to their networks in effect trusted that SolarWinds had a strong handle on their security posture. In Australia, CPS 234 mandates that APRA-regulated entities will need to have information security measures in place and includes cybersecurity assessments by independent assessors. Obviously, the effectiveness will boil down to the thoroughness of the assessor.

All this gets more complicated when you start to look into how the breach occurred in the first instance. It is starting to increasingly seem like hackers leveraged widely-used protocols and solutions. Ars Technica has a fascinating report referencing security firm Volexity, who encountered the same attackers in 2019; at the time, they bypassed MFA protections for Microsoft Outlook Web App (OWA) users.

Toward the end of the second incident that Volexity worked involving Dark Halo, the actor was observed accessing the e-mail account of a user via OWA. This was unexpected for a few reasons, not least of which was the targeted mailbox was protected by MFA. Logs from the Exchange server showed that the attacker provided username and password authentication like normal but were not challenged for a second factor through Duo. The logs from the Duo authentication server further showed that no attempts had been made to log into the account in question. Volexity was able to confirm that session hijacking was not involved and, through a memory dump of the OWA server, could also confirm that the attacker had presented cookie tied to a Duo MFA session named duo-sid.

Krebs on Security has a great write-up with a sobering quote from the DHS’s Cybersecurity and Infrastructure Security Agency.

CISA’s advisory specifically noted that “one of the principal ways the adversary is accomplishing this objective is by compromising the Security Assertion Markup Language (SAML) signing certificate using their escalated Active Directory privileges. Once this is accomplished, the adversary creates unauthorized but valid tokens and presents them to services that trust SAML tokens from the environment. These tokens can then be used to access resources in hosted environments, such as email, for data exfiltration via authorized application programming interfaces (APIs).”

The CISA goes on to advise that if an org identifies ‘SAML abuse’, mitigating individual issues might not be enough; you’ll need to consider the entire identity store as compromised. And unfortunately, the only remedy to that is building back identity and trust services from the ground up.

Additional Reading:

VMware Flaw a Vector in SolarWinds Breach?

SolarWinds hackers have a clever way to bypass multi-factor authentication

FireEye Threat Research

It Takes Two (To Thwart Data Breaches)

Some interesting insight from Gemalto's 2017 Data Breaches and Customer Loyalty Report:

• Of the 10,000 consumers interviewed, only 27% feel businesses take customer data security seriously
• 70% would take their business elsewhere following a breach
• 41% fail to take advantage of available security measures available such as multi-factor authentication
• 56% use the same password for multiple online accounts

While consumers are rightfully skeptical of the security hygiene of businesses they interact with, there is certainly a role for consumers to play here. 

Krebs on IoT Vulnerabilities

Brian Krebs has some interesting insight into this past weekend's DDoS attack on Dyn, an internet infrastructure company that provides services for some of the web's biggest destinations including Twitter, Amazon, Reddit and Netflix.

At first, it was unclear who or what was behind the attack on Dyn. But over the past few hours, at least one computer security firm has come out saying the attack involved Mirai, the same malware strain that was used in the record 620 Gpbs attack on my site last month. At the end September 2016, the hacker responsible for creating the Mirai malware released the source code for it, effectively letting anyone build their own attack army using Mirai.

Mirai scours the Web for IoT devices protected by little more than factory-default usernames and passwords, and then enlists the devices in attacks that hurl junk traffic at an online target until it can no longer accommodate legitimate visitors or users.

...

The wholesalers and retailers of these devices might then be encouraged to shift their focus toward buying and promoting connected devices which have this industry security association seal of approval. Consumers also would need to be educated to look for that seal of approval. Something like Underwriters Laboratories (UL), but for the Internet, perhaps.

Until then, these insecure IoT devices are going to stick around like a bad rash — unless and until there is a major, global effort to recall and remove vulnerable systems from the Internet. In my humble opinion, this global cleanup effort should be funded mainly by the companies that are dumping these cheap, poorly-secured hardware devices onto the market in an apparent bid to own the market. Well, they should be made to own the cleanup efforts as well.

The upside here is that IoT manufacturers and vendors will now have to wisen up to the fact that they have more to gain from secure devices and a lot to lose from a repeat of this weekend's events.