Internet Is Too Small To Feed AI Ambitions

"Companies racing to develop more powerful artificial intelligence are rapidly nearing a new problem: The internet might be too small for their plans.

Ever more robust systems developed by OpenAI, Google and others require larger oceans of information to learn from, straining the available pool of quality public data online at the same time that some data owners are blocking access to AI companies. Some executives and researchers say the industry's need for high-quality text data could outstrip supply within two years.

AI companies are hunting for untapped information sources, and rethinking how they train these systems. OpenAI, the maker of ChatGPT, has discussed training its next model on transcriptions of public YouTube videos, people familiar with the matter said.

Companies also are experimenting with using AI-generated, or synthetic, data as training material -- an approach many researchers say could cause crippling malfunctions.

These efforts are often secret because executives think solutions could be a competitive advantage.

"There is no established way of doing this," said Ari Morcos, an AI researcher who worked at Meta Platforms and Google's DeepMind before founding DatologyAI, whose backers include some AI pioneers. It builds tools to improve data selection, which could make training cheaper.

Data is among several essential AI resources in short supply. The chips needed to run so-called large-language models behind AI bots also are scarce. And industry leaders worry about a dearth of data centers and electricity to power them.

AI language models are built using text vacuumed up from the internet. That material is broken into tokens -- words and parts of words that the models use to learn how to formulate humanlike expressions. Generally, AI models become more capable the more data they train on.

Pablo Villalobos, who studies artificial intelligence for research institute Epoch, estimated that OpenAI's most advanced language model, called GPT-4, was trained on as many as 12 trillion tokens. An AI system like GPT-5 would need 60 trillion to 100 trillion on the current growth trajectory, Villalobos and other researchers have estimated. Harnessing all the high-quality data available could leave a shortfall of 10 trillion to 20 trillion tokens or more, he said.

Two years ago, Villalobos and his colleagues wrote that there was a 90% chance demand for high-quality data would outstrip supply by 2026. They have since become a bit more optimistic, and plan to update their estimate to 2028.

Most of the data available online is useless for AI training because it contains flaws such as sentence fragments or doesn't add to a model's knowledge. Villalobos estimated that the useful sliver is perhaps just one-tenth of the information gathered by Common Crawl, whose web archive is widely used by AI developers.

Meanwhile, social-media platforms, news publishers and others have been curbing access to data for AI training over concerns about issues including compensation. And there is little public will to hand over private conversational data, such as chats over iMessage.

Meta Platforms CEO Mark Zuckerberg recently touted his company's access to data on its platforms as a significant advantage in its AI efforts.

Some tech companies, including OpenAI partner Microsoft, are building language models that are a fraction of the size of GPT-4 but could accomplish specific objectives.

OpenAI Chief Executive Sam Altman has indicated the company is working on new training methods. "I think we're at the end of the era where it's going to be these giant, giant models," he said last year." [1]

1. Internet Is Too Small To Feed AI Ambitions. Seetharaman, Deepa.  Wall Street Journal, Eastern edition; New York, N.Y.. 02 Apr 2024: B.2.

 

Lenkijos pramonėje yra išskirtinai pastovus nuosmukis. Gamyba ir toliau mažėja

   „Lenkijos perdirbimo pramonėje gamyba kovo mėnesį sumažėjo jau 23-ią kartą iš eilės. Dėl silpnos paklausos ir besistabilizuojančių sąnaudų pramonės įmonės vėl sumažino jų prekių kainas.

 

     Tokį Lenkijos pramonės padėties vaizdą pateikia PMI – ekonominis rodiklis, kurį „S&P Global“ apskaičiavo, remdamasi maždaug 250 įmonių vadovų apklausa.

 

     Kovo mėnesį, kaip antradienį pranešė S&P, PMI padidėjo iki 48 punktų nuo 47,9 punkto vasarį. Anksčiau jis buvo didesnis praėjusių metų lapkritį. Šis minimalus padidėjimas atitinka „Parkiet“ apklaustų ekonomistų vertinimus, rodančius PMI stabilizavimąsi.

 

     PMI rodo, kad Lenkijos pramonėje yra nuolatinis nuosmukis

 

     Po antrojo iš eilės padidėjimo kovo mėn. PMI vis dar rodo, kad Lenkijos pramonė patiria itin pastovų nuosmukį. Jau 23-ią kartą iš eilės šis rodiklis nesiekia 50 punktų, o tai teoriškai reiškia, kad ekonominė situacija perdirbimo pramonėje kas mėnesį prastėja. Atstumas nuo šios sienos yra šio pablogėjimo greičio matas. Kovo mėnesį tai dar buvo pastebima. Be to, pagrindiniai PMI komponentai, atspindintys gamybos pokyčius ir naujus užsakymus, 25 mėnesius išliko žemiau 50 punktų – ilgiausiai šio rodiklio istorijoje, datuojami 1998 m.

 

     Žvelgiant į bendrą vaizdą, PMI išlieka šiek tiek didėjimo tendencija. Tuo tarpu šio rodiklio kitimo kryptis dažniausiai daugiau pasako apie ekonominę situaciją pramonėje, nei jo lygis. Per pastaruosius tris mėnesius PMI vidutiniškai siekė 47,7 punkto – aukščiausią lygį nuo 2023 m. balandžio mėn., neįskaitant sausio mėn. Tačiau išsamūs S&P tyrimo tarp įmonių rezultatai neteikia daug vilčių greitai pagerėti Lenkijos pramonės ekonominei situacijai.

 

     Lenkijos perdirbimo pramonės gamyba vasario mėnesį sumažėjo 22-ą kartą iš eilės. Tokio ilgo nuosmukio šiame sektoriuje nebuvo bent jau nuo 1998 m. Tačiau įmonės tampa vis optimistiškesnės.

 

     PMI skaičiuojamas pagal įmonių vadovų atsakymus į klausimus apie gamybos pokyčius (palyginti su praėjusiu mėnesiu), užsakymo vertę, užimtumą, pristatymo laiką ir atsargas.

 

       Naujausia apklausa rodo, kad kovo mėnesį gamintojai ir toliau kovojo su silpna paklausa, ypač iš Vokietijos ir Nyderlandų, dėl kurių buvo apribota gamyba.

 

     Tai savo ruožtu reiškė, kad ir vėl daugiau įmonių deklaravo užimtumo sumažėjimą, nei padidėjimą.

 

     Ekonominę situaciją euro zonos pramonėje matuojantis PMI nuo 46,6 punkto sausį nukrito iki 46,1 punkto vasarį ir tikimasi, kad jis padidės iki 47 punktų. Tačiau paslaugų sektoriaus PMI padidėjo nuo 48,4 punkto iki 50 punktų, t. y. iki lygio, skiriančio nuosmukį nuo plėtros.

 

     Silpna paklausa 12 mėn

 

     Dėl silpnos paklausos įmonės jau 12 mėnesį iš eilės siekė mažinti medžiagų ir komponentų atsargas. Tačiau kovą tai nesutrukdė didinti gatavos produkcijos atsargų, o tai kai kurie vadovai aiškino silpnesniais, nei tikėtasi, pardavimais. Tokiomis aplinkybėmis įmonės vėl sumažino jų prekių kainas, nors medžiagų sąnaudos šiek tiek išaugo. Toks vaizdas atitinka vidutinius „Parkiet“ apklaustų ekonomistų vertinimus, pagal kuriuos Lenkijoje parduotos pramonės produkcijos kainos kovo mėnesį per metus sumažėjo 9,5 proc., vasarį sumažinus 10,1 proc.

 

     S&P apklausti vadovai, vertindami jų įmonių perspektyvas, dažniau būna optimistiški, nei pesimistai. Daugiau jų tikisi, kad per ateinančius 12 mėnesių gamyba padidės, nei sumažės. Tačiau kovo mėnesį šio optimizmo mastas buvo šiek tiek mažesnis, nei ankstesnius du mėnesius“.

 

There is an exceptionally persistent recession in the Polish industry. Production continues to decline

"Production in the Polish processing industry decreased in March for the 23rd time in a row. Due to weak demand and stabilizing costs, industrial companies again reduced the prices of their goods.

This picture of the situation in the Polish industry is presented by PMI, an economic indicator calculated by S&P Global based on a survey among managers of approximately 250 enterprises.

In March, as S&P reported on Tuesday, the PMI increased to 48 points from 47.9 points in February. It was previously higher in November last year. This minimal increase is consistent with the estimates of economists surveyed by "Parkiet", which indicated a stabilization of the PMI.

PMI indicates that there is a persistent recession in Polish industry

After the second consecutive increase in March, the PMI still indicates that the Polish industry is in an extremely persistent recession. For the 23rd time in a row, this indicator is below 50 points, which theoretically means that the economic situation in the processing industry is deteriorating month by month. The distance from this border is a measure of the rate of this deterioration. In March it was still noticeable. Moreover, the main components of PMI, reflecting changes in production and new orders, have remained below 50 points for 25 months, the longest in the history of this indicator, dating back to 1998.

Looking at the bigger picture, PMI remains in a mild upward trend. Meanwhile, the direction of changes in this indicator usually says more about the economic situation in the industry than its level. Over the past three months, the PMI averaged 47.7 points, the highest since April 2023, excluding January. However, the detailed results of the S&P survey among companies do not give much hope for a quick improvement in the economic situation in the Polish industry.

Production in the Polish processing industry decreased in February for the 22nd time in a row. There hasn't been such a long decline in this sector since at least 1998. But companies are becoming more optimistic.

PMI is calculated based on company managers' answers to questions regarding changes (compared to the previous month) in production, order value, employment, delivery time and inventories.

  The latest survey shows that in March, manufacturers continued to struggle with weak demand, especially from Germany and the Netherlands, which forced production cuts. 

This, in turn, meant that once again more companies declared a decrease in employment than its increase, although the advantage of the former was among the smallest in the last two years.

The PMI measuring the economic situation in the euro zone industry dropped from 46.6 points in January to 46.1 points in February, and an increase to 47 points was expected. However, the PMI for the services sector increased from 48.4 points to 50 points, i.e. to the level separating recession from expansion.

Weak demand for 12 months

Due to weak demand, companies sought to reduce inventories of materials and components for the 12th consecutive month. In March, however, this did not prevent an increase in stocks of finished products, which some managers explained by weaker-than-expected sales. In such circumstances, companies again reduced the prices of their goods, even though the costs of materials increased slightly. This picture is consistent with the average estimates of economists surveyed by "Parkiet", according to which the prices of sold industrial production in Poland decreased by 9.5 percent in March year on year after a 10.1% discount in February.

When assessing the prospects of their companies, managers surveyed by S&P are more often optimistic than pessimistic. More of them expect an increase in production in the next 12 months than a decrease. But in March, the scale of this optimism was slightly smaller than in the previous two months."