[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"summary-9d8182e1f5a5347f-fixing-computer-use-benchmarks-beyond-replay-explo-summary":3,"summaries-facets-categories":141,"summary-related-9d8182e1f5a5347f-fixing-computer-use-benchmarks-beyond-replay-explo-summary":7045},{"id":4,"title":5,"ai":6,"body":13,"categories":96,"created_at":98,"date_modified":98,"description":90,"extension":99,"faq":98,"featured":100,"kicker_label":98,"meta":101,"navigation":120,"path":121,"published_at":122,"question":98,"scraped_at":123,"seo":124,"sitemap":125,"source_id":126,"source_name":127,"source_type":128,"source_url":129,"stem":130,"tags":131,"thumbnail_url":136,"tldr":137,"tweet":138,"unknown_tags":139,"__hash__":140},"summaries\u002Fsummaries\u002F9d8182e1f5a5347f-fixing-computer-use-benchmarks-beyond-replay-explo-summary.md","Fixing Computer Use Benchmarks: Beyond Replay Exploits",{"provider":7,"model":8,"input_tokens":9,"output_tokens":10,"processing_time_ms":11,"cost_usd":12},"openrouter","google\u002Fgemini-3.1-flash-lite",7094,696,3083,0.0028175,{"type":14,"value":15,"toc":89},"minimark",[16,21,30,34,37,72,79,83,86],[17,18,20],"h2",{"id":19},"the-replay-agent-exploit","The 'Replay Agent' Exploit",[22,23,24,25,29],"p",{},"Many current computer use benchmarks are deterministic, allowing them to be gamed by \"replay agents.\" These are scripts—often under one megabyte—that simply record a successful trajectory once and replay it blindly. Because these benchmarks lack variation, these scripts frequently match or outperform the frontier models they were derived from. The metric ",[26,27,28],"code",{},"pass@k"," is particularly problematic in these environments, as it effectively functions as a formal measure of this replay exploit rather than a measure of agent intelligence.",[17,31,33],{"id":32},"the-prism-principles-for-robust-environments","The PRISM Principles for Robust Environments",[22,35,36],{},"To move beyond static, hackable benchmarks, environments must be designed with the PRISM principles:",[38,39,40,48,54,60,66],"ul",{},[41,42,43,47],"li",{},[44,45,46],"strong",{},"Privileged Verification:"," Use ground-truth data to verify outcomes.",[41,49,50,53],{},[44,51,52],{},"Realism:"," Ensure the environment is a faithful reproduction of real-world systems.",[41,55,56,59],{},[44,57,58],{},"Integrity-Checked Configurations:"," Validate that every generated variation is functional.",[41,61,62,65],{},[44,63,64],{},"Sandboxed Execution:"," Isolate the agent's actions.",[41,67,68,71],{},[44,69,70],{},"Multifactorial Variation:"," Introduce stochasticity across data, themes, and starting states.",[22,73,74,75,78],{},"Pierluca D'Oro introduces ",[44,76,77],{},"DIGIWORLD"," as an implementation of these principles, featuring 15 mobile apps and 3.2 million verified configurations. The key to scaling this is a compiler-like system that generates combinations and automatically rejects broken configurations, ensuring that the benchmark remains both diverse and valid.",[17,80,82],{"id":81},"honest-uncertainty-and-financial-impact","Honest Uncertainty and Financial Impact",[22,84,85],{},"Even with a robust environment, evaluation metrics often fail due to \"overconfident\" statistics. Naive rollouts on a single base case typically yield confidence intervals that cover the true performance only 20% of the time, rather than the expected 95%.",[22,87,88],{},"This lack of rigor has direct financial consequences. In a scenario with one million tasks, a 4% performance gap between two models—hidden by tight but inaccurate confidence intervals—can result in hundreds of thousands of dollars in losses per month. A rigorous evaluation methodology must account for two sources of variance: the model's stochastic actions and the environment's inherent variability. When uncertainty is measured honestly, the data may reveal that a model's performance is statistically indistinguishable from another, preventing costly, premature deployment decisions.",{"title":90,"searchDepth":91,"depth":91,"links":92},"",2,[93,94,95],{"id":19,"depth":91,"text":20},{"id":32,"depth":91,"text":33},{"id":81,"depth":91,"text":82},[97],"AI & LLMs",null,"md",false,{"content_references":102,"triage":115},[103,109,113],{"type":104,"title":105,"author":106,"url":107,"context":108},"paper","Computer Use at the Edge of the Statistical Precipice","Pierluca D'Oro et al.","https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.08261","reviewed",{"type":110,"title":111,"context":112},"tool","OSWorld","mentioned",{"type":110,"title":77,"context":114},"recommended",{"relevance":116,"novelty":117,"quality":116,"actionability":117,"composite":118,"reasoning":119},4,3,3.6,"Category: Software Engineering. The article discusses the limitations of current benchmarking methods in AI evaluation, addressing a specific pain point regarding the reliability of performance metrics. It introduces the PRISM principles for creating robust environments, which provides actionable insights, though it lacks detailed implementation steps.",true,"\u002Fsummaries\u002F9d8182e1f5a5347f-fixing-computer-use-benchmarks-beyond-replay-explo-summary","2026-08-14 14:30:31","2026-08-15 03:10:41",{"title":5,"description":90},{"loc":121},"9d8182e1f5a5347f","AI Engineer","video","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=CTLa_p6iOiY","summaries\u002F9d8182e1f5a5347f-fixing-computer-use-benchmarks-beyond-replay-explo-summary",[132,133,134,135],"ai-agents","benchmarking","evaluation","software-engineering","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FCTLa_p6iOiY\u002Fhqdefault.jpg","Current computer use benchmarks are often gamed by 'replay agents' that blindly repeat successful trajectories. Robust evaluation requires stochastic, verified environments and honest statistical uncertainty to avoid costly deployment errors.","This video explains how current computer-use benchmarks are easily gamed by \"replay agents\"—scripts that simply record and repeat successful trajectories—and introduces [PRISM](https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.08261) principles to build more robust, non-deterministic evaluation environments like [DIGIWORLD](https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.08261).",[132,133,134,135],"JBQjX97nRzWs5zMXjQgY9BeQPq24pI-78OCykeA1Gcc",[142,144,147,149,152,154,157,160,162,164,166,168,171,173,175,177,179,182,184,186,188,190,193,196,198,200,202,204,206,208,210,212,214,216,218,220,222,224,226,228,230,232,234,236,238,240,242,245,247,249,251,253,255,257,259,261,263,265,267,269,271,274,276,278,280,282,284,286,288,290,292,294,296,298,300,302,304,306,308,310,312,315,317,319,321,323,325,327,329,331,333,335,337,339,341,344,346,348,350,352,354,356,358,360,362,364,366,368,370,372,374,376,378,380,382,384,386,388,390,392,394,396,398,400,402,404,406,409,411,413,415,417,419,421,423,425,427,429,432,434,436,438,440,442,444,446,448,450,452,454,456,458,460,462,465,467,469,471,473,475,477,479,481,483,485,488,490,492,494,496,498,500,502,504,506,508,510,512,514,516,518,520,522,524,526,528,530,532,534,536,538,540,542,544,547,549,551,554,556,558,560,562,564,566,568,570,572,574,576,578,580,582,584,586,588,590,593,595,597,599,601,603,605,607,609,611,613,615,618,620,622,624,626,628,630,632,634,636,638,640,642,644,646,648,650,652,654,656,658,660,662,664,666,668,670,672,674,676,678,680,682,684,686,688,690,692,694,696,698,700,702,704,706,708,710,712,714,716,718,720,722,724,726,728,730,732,734,736,738,740,742,744,746,748,750,752,754,756,758,760,762,764,766,768,770,772,774,776,778,780,782,784,786,788,790,792,794,796,798,800,802,804,806,808,810,812,814,816,818,820,822,824,826,828,830,832,834,836,838,840,842,844,846,848,850,852,854,856,858,860,862,864,866,868,870,872,874,876,878,880,882,884,886,888,890,892,894,896,898,900,902,905,907,909,911,913,916,918,920,922,924,926,928,930,932,934,936,938,940,943,945,947,949,951,953,955,957,959,961,963,965,967,969,971,973,975,977,979,981,983,985,987,989,991,993,995,997,999,1001,1003,1005,1007,1009,1011,1013,1015,1017,1019,1021,1023,1025,1027,1029,1031,1033,1035,1037,1039,1041,1043,1045,1047,1049,1051,1053,1055,1057,1059,1061,1063,1065,1067,1069,1071,1073,1075,1077,1079,1081,1083,1085,1087,1089,1091,1093,1095,1097,1099,1101,1103,1105,1107,1109,1111,1113,1115,1117,1119,1121,1123,1125,1127,1129,1131,1133,1135,1137,1139,1141,1143,1145,1147,1149,1151,1153,1155,1157,1159,1161,1163,1165,1167,1169,1171,1173,1175,1177,1179,1181,1183,1185,1187,1189,1191,1193,1195,1197,1199,1201,1203,1205,1207,1209,1211,1213,1215,1217,1219,1221,1223,1225,1227,1229,1231,1233,1235,1237,1239,1241,1243,1245,1247,1249,1251,1253,1255,1257,1259,1261,1263,1265,1268,1270,1272,1274,1276,1278,1280,1282,1284,1286,1288,1290,1292,1294,1296,1298,1300,1302,1304,1306,1308,1310,1312,1314,1316,1318,1320,1322,1324,1326,1328,1330,1332,1334,1336,1338,1340,1342,1344,1346,1348,1350,1352,1354,1356,1358,1360,1362,1364,1366,1368,1370,1372,1374,1376,1378,1380,1382,1384,1386,1388,1390,1392,1394,1396,1398,1400,1402,1404,1406,1408,1410,1412,1414,1416,1418,1420,1422,1424,1426,1428,1430,1432,1434,1436,1438,1440,1442,1444,1446,1448,1450,1452,1454,1456,1458,1460,1462,1464,1466,1469,1471,1473,1475,1477,1479,1481,1483,1485,1487,1489,1491,1493,1495,1497,1499,1501,1503,1505,1507,1509,1511,1513,1515,1517,1519,1521,1523,1525,1527,1529,1531,1533,1535,1537,1539,1541,1543,1545,1547,1549,1551,1553,1555,1557,1559,1561,1563,1565,1567,1569,1571,1573,1575,1577,1579,1581,1583,1585,1587,1589,1591,1593,1595,1597,1599,1601,1603,1605,1608,1610,1612,1614,1616,1618,1620,1622,1624,1626,1628,1630,1632,1634,1636,1638,1640,1642,1644,1646,1648,1650,1652,1654,1656,1658,1660,1662,1664,1666,1668,1670,1672,1674,1676,1678,1680,1682,1684,1686,1688,1690,1692,1694,1696,1698,1700,1702,1704,1706,1708,1710,1712,1714,1716,1718,1720,1722,1724,1726,1728,1730,1732,1734,1736,1738,1740,1742,1744,1746,1748,1750,1752,1754,1756,1758,1760,1762,1764,1767,1769,1771,1773,1775,1777,1779,1781,1783,1785,1787,1789,1791,1793,1795,1797,1799,1801,1803,1805,1807,1809,1811,1813,1815,1817,1819,1821,1823,1825,1827,1830,1832,1834,1836,1838,1840,1842,1844,1846,1848,1850,1852,1854,1856,1858,1860,1862,1864,1866,1868,1870,1872,1874,1876,1878,1880,1882,1884,1886,1888,1890,1892,1894,1896,1898,1900,1902,1904,1906,1908,1910,1912,1914,1916,1918,1920,1922,1924,1926,1928,1930,1932,1934,1936,1938,1940,1942,1944,1946,1948,1950,1952,1954,1956,1958,1960,1962,1964,1966,1968,1970,1972,1974,1976,1978,1980,1982,1984,1986,1988,1990,1992,1994,1996,1998,2000,2002,2004,2006,2008,2010,2012,2014,2016,2018,2020,2022,2024,2026,2028,2030,2032,2034,2036,2038,2040,2042,2044,2046,2048,2050,2052,2054,2056,2058,2060,2062,2064,2066,2068,2070,2072,2074,2076,2078,2080,2082,2084,2086,2088,2090,2092,2094,2096,2098,2100,2102,2104,2106,2108,2110,2112,2114,2116,2118,2120,2122,2124,2126,2128,2130,2132,2134,2136,2138,2140,2142,2144,2146,2148,2150,2152,2154,2156,2158,2160,2162,2164,2166,2168,2170,2172,2174,2176,2178,2180,2182,2184,2186,2188,2190,2192,2194,2196,2198,2200,2202,2204,2206,2208,2210,2212,2214,2216,2218,2220,2222,2224,2226,2228,2230,2232,2234,2236,2238,2240,2242,2244,2246,2248,2250,2252,2254,2256,2258,2260,2262,2264,2266,2268,2271,2273,2275,2277,2279,2281,2283,2285,2287,2289,2291,2293,2295,2297,2299,2301,2303,2305,2307,2309,2311,2313,2315,2317,2319,2321,2323,2325,2327,2329,2331,2333,2335,2337,2339,2341,2343,2345,2347,2349,2351,2353,2355,2357,2359,2361,2364,2366,2368,2370,2372,2374,2376,2378,2380,2382,2384,2386,2388,2390,2392,2394,2396,2398,2400,2402,2404,2406,2408,2410,2412,2414,2416,2418,2420,2422,2424,2426,2428,2430,2432,2434,2436,2438,2440,2442,2444,2446,2448,2450,2452,2454,2456,2458,2460,2462,2464,2466,2468,2470,2472,2474,2476,2478,2480,2483,2485,2487,2489,2491,2493,2495,2497,2499,2501,2503,2505,2507,2509,2511,2513,2515,2517,2519,2521,2523,2526,2528,2530,2532,2534,2536,2538,2540,2542,2544,2546,2548,2550,2552,2554,2556,2558,2560,2562,2564,2566,2568,2570,2572,2574,2576,2578,2580,2582,2584,2586,2588,2590,2592,2594,2596,2598,2600,2602,2604,2606,2608,2610,2612,2614,2616,2618,2620,2622,2624,2626,2628,2630,2632,2634,2636,2638,2640,2642,2644,2646,2648,2650,2652,2654,2656,2658,2660,2662,2664,2666,2668,2670,2672,2674,2676,2678,2680,2682,2684,2686,2688,2690,2692,2694,2696,2698,2700,2702,2704,2706,2708,2710,2712,2714,2716,2718,2720,2722,2724,2726,2728,2730,2732,2734,2736,2738,2740,2742,2744,2746,2748,2750,2752,2754,2756,2758,2760,2762,2764,2766,2768,2770,2772,2774,2776,2778,2780,2782,2784,2786,2788,2790,2792,2794,2796,2798,2800,2802,2804,2806,2808,2810,2812,2814,2816,2818,2820,2822,2824,2826,2828,2830,2832,2834,2836,2838,2840,2842,2844,2846,2848,2850,2852,2854,2856,2858,2860,2862,2864,2866,2868,2870,2872,2874,2876,2878,2880,2882,2884,2886,2888,2890,2892,2894,2896,2898,2900,2902,2904,2906,2908,2910,2912,2914,2916,2918,2920,2922,2924,2926,2928,2930,2932,2934,2936,2938,2940,2942,2944,2946,2948,2950,2952,2954,2956,2958,2960,2962,2964,2966,2968,2970,2972,2974,2976,2978,2980,2982,2984,2986,2988,2990,2992,2994,2996,2998,3000,3002,3004,3006,3008,3010,3012,3014,3016,3018,3020,3022,3024,3026,3028,3030,3032,3034,3036,3038,3040,3042,3044,3046,3048,3050,3052,3054,3056,3058,3060,3062,3064,3066,3068,3070,3072,3074,3076,3078,3080,3082,3084,3086,3088,3090,3092,3094,3096,3098,3100,3102,3104,3106,3108,3110,3112,3114,3116,3118,3120,3122,3124,3126,3128,3130,3132,3134,3136,3138,3140,3142,3144,3146,3148,3150,3152,3154,3156,3158,3160,3162,3164,3167,3169,3171,3173,3175,3177,3179,3181,3183,3185,3187,3189,3191,3193,3195,3197,3199,3201,3203,3205,3207,3209,3211,3213,3215,3217,3219,3221,3223,3225,3227,3229,3231,3233,3235,3237,3239,3241,3243,3245,3247,3249,3251,3253,3255,3257,3259,3261,3263,3265,3267,3269,3271,3273,3275,3277,3279,3281,3283,3285,3287,3289,3291,3294,3296,3298,3300,3302,3304,3306,3308,3310,3312,3314,3316,3318,3320,3322,3324,3326,3328,3330,3332,3334,3336,3338,3340,3342,3344,3346,3348,3350,3352,3354,3356,3358,3360,3362,3364,3366,3368,3370,3372,3374,3376,3378,3380,3382,3384,3386,3388,3390,3392,3394,3396,3398,3400,3402,3404,3406,3408,3410,3412,3414,3416,3418,3420,3422,3424,3426,3428,3430,3432,3434,3436,3438,3440,3442,3444,3446,3448,3450,3452,3454,3456,3458,3460,3462,3464,3466,3468,3470,3472,3474,3476,3478,3480,3482,3484,3486,3488,3490,3492,3494,3496,3498,3500,3502,3504,3506,3508,3510,3512,3514,3516,3518,3520,3522,3524,3526,3528,3530,3532,3534,3536,3538,3540,3542,3544,3546,3548,3550,3552,3554,3556,3558,3560,3562,3564,3566,3568,3570,3572,3574,3576,3578,3580,3582,3584,3586,3588,3590,3592,3594,3596,3598,3600,3602,3604,3606,3608,3610,3612,3614,3616,3618,3620,3622,3624,3626,3628,3630,3632,3634,3636,3638,3640,3642,3644,3646,3648,3650,3652,3654,3656,3658,3660,3662,3664,3666,3668,3670,3672,3674,3676,3678,3680,3682,3684,3686,3688,3690,3692,3694,3696,3698,3700,3702,3704,3706,3708,3710,3712,3714,3716,3718,3720,3722,3724,3726,3728,3730,3732,3734,3736,3738,3740,3742,3744,3746,3748,3750,3752,3754,3756,3758,3760,3762,3764,3766,3768,3770,3772,3774,3776,3778,3780,3782,3784,3786,3788,3790,3792,3794,3796,3798,3800,3802,3804,3806,3808,3810,3812,3814,3816,3818,3820,3822,3824,3826,3828,3830,3832,3834,3836,3838,3840,3842,3844,3846,3848,3850,3852,3854,3856,3858,3860,3862,3864,3866,3868,3870,3872,3874,3876,3878,3880,3882,3884,3886,3888,3890,3892,3894,3896,3898,3900,3902,3904,3906,3908,3910,3912,3914,3916,3918,3920,3922,3924,3926,3928,3930,3932,3934,3936,3938,3940,3942,3944,3946,3948,3950,3952,3954,3956,3958,3960,3962,3964,3966,3968,3970,3972,3974,3976,3978,3980,3982,3984,3986,3988,3990,3992,3994,3996,3998,4000,4002,4004,4006,4008,4010,4012,4014,4016,4018,4020,4022,4024,4026,4028,4030,4032,4034,4036,4038,4040,4042,4044,4046,4048,4050,4052,4054,4056,4058,4060,4062,4064,4066,4068,4070,4072,4074,4076,4078,4080,4082,4084,4086,4088,4090,4092,4094,4096,4098,4100,4102,4104,4106,4108,4110,4112,4114,4116,4118,4120,4122,4124,4126,4128,4130,4132,4134,4136,4138,4140,4142,4144,4146,4148,4150,4152,4154,4156,4158,4160,4162,4164,4166,4168,4170,4172,4174,4176,4178,4180,4182,4184,4186,4188,4190,4192,4194,4196,4198,4200,4202,4204,4206,4208,4210,4212,4214,4216,4218,4220,4222,4224,4226,4228,4230,4232,4234,4236,4238,4240,4242,4244,4246,4248,4250,4252,4254,4256,4258,4260,4262,4264,4266,4268,4270,4272,4274,4276,4278,4280,4282,4284,4286,4288,4290,4292,4294,4296,4298,4300,4302,4304,4306,4308,4310,4312,4314,4316,4318,4320,4322,4324,4326,4328,4330,4332,4334,4336,4338,4340,4342,4344,4346,4348,4350,4352,4354,4356,4358,4360,4362,4364,4366,4368,4370,4372,4374,4376,4378,4380,4382,4384,4386,4388,4390,4392,4394,4396,4398,4400,4402,4404,4406,4408,4410,4412,4414,4416,4418,4420,4422,4424,4426,4428,4430,4432,4434,4436,4438,4440,4442,4444,4446,4448,4450,4452,4454,4456,4458,4460,4462,4464,4466,4468,4470,4472,4474,4476,4478,4480,4482,4484,4486,4488,4490,4492,4494,4496,4498,4500,4502,4504,4506,4508,4510,4512,4514,4516,4518,4520,4522,4524,4526,4528,4530,4532,4534,4536,4538,4540,4542,4544,4546,4548,4550,4552,4554,4556,4558,4560,4562,4564,4566,4568,4570,4572,4574,4576,4579,4581,4583,4585,4587,4589,4591,4593,4595,4597,4599,4601,4603,4605,4607,4609,4611,4613,4615,4617,4619,4621,4623,4625,4627,4629,4631,4633,4635,4637,4639,4641,4643,4645,4647,4649,4651,4653,4655,4657,4659,4661,4663,4665,4667,4669,4671,4673,4675,4677,4679,4681,4683,4685,4687,4689,4691,4693,4695,4697,4699,4701,4703,4705,4707,4709,4711,4713,4715,4717,4719,4721,4723,4725,4727,4729,4731,4733,4735,4737,4739,4741,4743,4745,4747,4749,4751,4753,4755,4757,4759,4761,4763,4765,4767,4769,4771,4773,4775,4777,4779,4781,4783,4785,4787,4789,4791,4793,4795,4797,4799,4801,4803,4805,4807,4809,4811,4813,4815,4817,4819,4821,4823,4825,4827,4829,4831,4833,4835,4837,4839,4841,4843,4845,4847,4849,4851,4853,4855,4857,4859,4861,4863,4865,4867,4869,4871,4873,4875,4877,4879,4881,4883,4885,4887,4889,4891,4893,4895,4897,4899,4901,4903,4905,4907,4909,4911,4913,4915,4917,4919,4921,4923,4925,4927,4929,4931,4933,4935,4937,4939,4941,4943,4945,4947,4949,4951,4953,4955,4957,4959,4961,4963,4965,4967,4969,4971,4973,4975,4977,4979,4981,4983,4985,4987,4989,4991,4993,4995,4997,4999,5001,5003,5005,5007,5009,5011,5013,5015,5017,5019,5021,5023,5025,5027,5029,5031,5033,5035,5037,5039,5041,5043,5045,5047,5049,5051,5053,5055,5057,5059,5061,5063,5065,5067,5069,5071,5073,5075,5077,5079,5081,5083,5085,5087,5089,5091,5093,5095,5097,5099,5101,5103,5105,5107,5109,5111,5113,5115,5117,5119,5121,5123,5125,5127,5129,5131,5133,5135,5137,5139,5141,5143,5145,5147,5149,5151,5153,5155,5157,5159,5161,5163,5165,5167,5169,5171,5173,5175,5177,5179,5181,5183,5185,5187,5189,5191,5193,5195,5197,5199,5201,5203,5205,5207,5209,5211,5213,5215,5217,5219,5221,5223,5225,5227,5229,5231,5233,5235,5237,5239,5241,5243,5245,5247,5249,5251,5253,5255,5257,5259,5261,5263,5265,5267,5269,5271,5273,5275,5277,5279,5281,5283,5285,5287,5289,5291,5293,5295,5297,5299,5301,5303,5305,5307,5309,5311,5313,5315,5317,5319,5321,5323,5325,5327,5329,5331,5333,5335,5337,5339,5341,5343,5345,5347,5349,5351,5353,5355,5357,5359,5361,5363,5365,5367,5369,5371,5373,5375,5377,5379,5381,5383,5385,5387,5389,5391,5393,5395,5397,5399,5401,5403,5405,5407,5409,5411,5413,5415,5417,5419,5421,5423,5425,5427,5429,5431,5433,5436,5438,5440,5442,5444,5446,5448,5450,5452,5454,5456,5458,5460,5462,5464,5466,5468,5470,5472,5474,5476,5478,5480,5482,5484,5486,5488,5490,5492,5494,5496,5498,5500,5502,5504,5506,5508,5510,5512,5514,5516,5518,5520,5522,5524,5526,5528,5530,5532,5534,5536,5538,5540,5542,5544,5546,5548,5550,5552,5554,5556,5558,5560,5562,5564,5566,5568,5570,5572,5574,5576,5578,5580,5582,5584,5586,5588,5590,5592,5594,5596,5598,5600,5602,5604,5606,5608,5610,5612,5614,5616,5618,5620,5622,5624,5626,5628,5630,5632,5634,5636,5639,5641,5643,5645,5647,5649,5651,5653,5655,5657,5659,5661,5663,5665,5667,5669,5671,5673,5675,5677,5679,5681,5683,5685,5687,5689,5691,5693,5695,5697,5699,5701,5703,5705,5707,5709,5711,5713,5715,5717,5719,5721,5723,5725,5727,5729,5731,5733,5735,5737,5739,5741,5743,5745,5747,5749,5751,5753,5755,5757,5759,5761,5763,5765,5767,5769,5771,5773,5775,5777,5779,5781,5783,5785,5787,5789,5791,5793,5795,5797,5799,5801,5803,5805,5807,5809,5811,5813,5815,5817,5819,5821,5823,5825,5827,5829,5831,5833,5835,5837,5839,5841,5843,5845,5847,5849,5851,5853,5855,5857,5859,5861,5863,5865,5867,5869,5871,5873,5875,5877,5879,5881,5883,5885,5887,5889,5891,5893,5895,5897,5899,5901,5903,5905,5907,5909,5911,5913,5915,5917,5919,5921,5923,5925,5927,5929,5931,5933,5935,5937,5939,5941,5943,5945,5947,5949,5951,5953,5955,5957,5959,5961,5963,5965,5967,5969,5971,5973,5975,5977,5979,5981,5983,5985,5987,5989,5991,5993,5995,5997,5999,6001,6003,6005,6007,6009,6011,6013,6015,6017,6019,6021,6023,6025,6027,6029,6031,6033,6035,6037,6039,6041,6043,6045,6047,6049,6051,6053,6055,6057,6059,6061,6063,6065,6067,6069,6071,6073,6075,6077,6079,6081,6083,6085,6087,6089,6091,6093,6095,6097,6099,6101,6103,6105,6107,6109,6111,6113,6115,6117,6119,6121,6123,6125,6127,6129,6131,6133,6135,6137,6139,6141,6143,6145,6147,6149,6151,6153,6155,6157,6159,6161,6163,6165,6167,6169,6171,6173,6175,6177,6179,6181,6183,6185,6187,6189,6191,6193,6195,6197,6199,6201,6203,6205,6207,6209,6211,6213,6215,6217,6219,6221,6223,6225,6227,6229,6231,6233,6235,6237,6239,6241,6243,6245,6247,6249,6251,6253,6255,6257,6259,6261,6263,6265,6267,6269,6271,6273,6275,6277,6279,6281,6283,6285,6287,6289,6291,6293,6295,6297,6299,6301,6303,6305,6307,6309,6311,6313,6315,6317,6319,6321,6323,6325,6327,6329,6331,6333,6335,6337,6339,6341,6343,6345,6347,6349,6351,6353,6355,6357,6359,6361,6363,6365,6367,6369,6371,6373,6375,6377,6379,6381,6383,6385,6387,6389,6391,6393,6395,6397,6399,6401,6403,6405,6407,6409,6411,6413,6415,6417,6419,6421,6423,6425,6427,6429,6431,6433,6435,6437,6439,6441,6443,6445,6447,6449,6451,6453,6455,6457,6459,6461,6463,6465,6467,6469,6471,6473,6475,6477,6479,6481,6483,6485,6487,6489,6491,6493,6495,6497,6499,6501,6503,6505,6507,6509,6511,6513,6515,6517,6519,6521,6523,6525,6527,6529,6531,6533,6535,6537,6539,6541,6543,6545,6547,6549,6551,6553,6555,6557,6559,6561,6563,6565,6567,6569,6571,6573,6575,6577,6579,6581,6583,6585,6587,6589,6591,6593,6595,6597,6599,6601,6603,6605,6607,6609,6611,6613,6615,6617,6619,6621,6623,6625,6627,6629,6631,6633,6635,6637,6639,6641,6643,6645,6647,6649,6651,6653,6655,6657,6659,6661,6663,6665,6667,6669,6671,6673,6675,6677,6679,6681,6683,6685,6687,6689,6691,6693,6695,6697,6699,6701,6703,6705,6707,6709,6711,6713,6715,6717,6719,6721,6723,6725,6727,6729,6731,6733,6735,6737,6739,6741,6743,6745,6747,6749,6751,6753,6755,6757,6759,6761,6763,6765,6767,6769,6771,6773,6775,6777,6779,6781,6783,6785,6787,6789,6791,6793,6795,6797,6799,6801,6803,6805,6807,6809,6811,6813,6815,6817,6819,6821,6823,6825,6827,6829,6831,6833,6835,6837,6839,6841,6843,6845,6847,6849,6851,6853,6855,6857,6859,6861,6863,6865,6867,6869,6871,6873,6875,6877,6879,6881,6883,6885,6887,6889,6891,6893,6895,6897,6899,6901,6903,6905,6907,6909,6911,6913,6915,6917,6919,6921,6923,6925,6927,6929,6931,6933,6935,6937,6939,6941,6943,6945,6947,6949,6951,6953,6955,6957,6959,6961,6963,6965,6967,6969,6971,6973,6975,6977,6979,6981,6983,6985,6987,6989,6991,6993,6995,6997,6999,7001,7003,7005,7007,7009,7011,7013,7015,7017,7019,7021,7023,7025,7027,7029,7031,7033,7035,7037,7039,7041,7043],{"categories":143},[97],{"categories":145},[146],"Developer Productivity",{"categories":148},[97],{"categories":150},[151],"Business & SaaS",{"categories":153},[97],{"categories":155},[156],"AI Automation",{"categories":158},[159],"Product Strategy",{"categories":161},[156],{"categories":163},[97],{"categories":165},[146],{"categories":167},[156],{"categories":169},[170],"Software Engineering",{"categories":172},[97],{"categories":174},[151],{"categories":176},[],{"categories":178},[97],{"categories":180},[181],"Inference & Serving",{"categories":183},[97],{"categories":185},[97],{"categories":187},[156],{"categories":189},[],{"categories":191},[192],"AI News & Trends",{"categories":194},[195],"Data Science & Visualization",{"categories":197},[156],{"categories":199},[97],{"categories":201},[97],{"categories":203},[151],{"categories":205},[146],{"categories":207},[97],{"categories":209},[156],{"categories":211},[192],{"categories":213},[97],{"categories":215},[156],{"categories":217},[156],{"categories":219},[97],{"categories":221},[97],{"categories":223},[156],{"categories":225},[97],{"categories":227},[97],{"categories":229},[97],{"categories":231},[156],{"categories":233},[192],{"categories":235},[97],{"categories":237},[97],{"categories":239},[97],{"categories":241},[],{"categories":243},[244],"Design & Frontend",{"categories":246},[195],{"categories":248},[192],{"categories":250},[97],{"categories":252},[97],{"categories":254},[97],{"categories":256},[],{"categories":258},[97],{"categories":260},[97],{"categories":262},[156],{"categories":264},[170],{"categories":266},[97],{"categories":268},[156],{"categories":270},[97],{"categories":272},[273],"Marketing & Growth",{"categories":275},[244],{"categories":277},[97],{"categories":279},[156],{"categories":281},[97],{"categories":283},[97],{"categories":285},[170],{"categories":287},[97],{"categories":289},[],{"categories":291},[],{"categories":293},[244],{"categories":295},[97],{"categories":297},[156],{"categories":299},[146],{"categories":301},[170],{"categories":303},[156],{"categories":305},[244],{"categories":307},[159],{"categories":309},[97],{"categories":311},[170],{"categories":313},[314],"DevOps & Cloud",{"categories":316},[156],{"categories":318},[159],{"categories":320},[192],{"categories":322},[97],{"categories":324},[],{"categories":326},[97],{"categories":328},[97],{"categories":330},[],{"categories":332},[156],{"categories":334},[170],{"categories":336},[],{"categories":338},[170],{"categories":340},[97],{"categories":342},[343],"Governance & Standards",{"categories":345},[151],{"categories":347},[],{"categories":349},[],{"categories":351},[97],{"categories":353},[97],{"categories":355},[156],{"categories":357},[97],{"categories":359},[97],{"categories":361},[156],{"categories":363},[97],{"categories":365},[97],{"categories":367},[97],{"categories":369},[],{"categories":371},[170],{"categories":373},[],{"categories":375},[],{"categories":377},[97],{"categories":379},[170],{"categories":381},[],{"categories":383},[170],{"categories":385},[97],{"categories":387},[97],{"categories":389},[273],{"categories":391},[97],{"categories":393},[97],{"categories":395},[97],{"categories":397},[244],{"categories":399},[244],{"categories":401},[97],{"categories":403},[170],{"categories":405},[156],{"categories":407},[408],"GovTech & Public-Sector Adoption",{"categories":410},[170],{"categories":412},[97],{"categories":414},[97],{"categories":416},[97],{"categories":418},[156],{"categories":420},[156],{"categories":422},[195],{"categories":424},[97],{"categories":426},[192],{"categories":428},[156],{"categories":430},[431],"Legal AI Tools",{"categories":433},[97],{"categories":435},[156],{"categories":437},[97],{"categories":439},[273],{"categories":441},[156],{"categories":443},[159],{"categories":445},[97],{"categories":447},[170],{"categories":449},[408],{"categories":451},[],{"categories":453},[156],{"categories":455},[],{"categories":457},[151],{"categories":459},[156],{"categories":461},[156],{"categories":463},[464],"RAG & Retrieval",{"categories":466},[151],{"categories":468},[97],{"categories":470},[170],{"categories":472},[170],{"categories":474},[314],{"categories":476},[244],{"categories":478},[156],{"categories":480},[97],{"categories":482},[97],{"categories":484},[],{"categories":486},[487],"Agents & Orchestration",{"categories":489},[170],{"categories":491},[97],{"categories":493},[],{"categories":495},[156],{"categories":497},[151],{"categories":499},[],{"categories":501},[97],{"categories":503},[],{"categories":505},[97],{"categories":507},[146],{"categories":509},[170],{"categories":511},[151],{"categories":513},[97],{"categories":515},[156],{"categories":517},[97],{"categories":519},[192],{"categories":521},[97],{"categories":523},[],{"categories":525},[97],{"categories":527},[],{"categories":529},[97],{"categories":531},[170],{"categories":533},[97],{"categories":535},[156],{"categories":537},[195],{"categories":539},[],{"categories":541},[97],{"categories":543},[244],{"categories":545},[546],"Models & Frontier Labs",{"categories":548},[],{"categories":550},[244],{"categories":552},[553],"Regulation & Governance of AI",{"categories":555},[159],{"categories":557},[156],{"categories":559},[],{"categories":561},[97],{"categories":563},[97],{"categories":565},[156],{"categories":567},[156],{"categories":569},[192],{"categories":571},[97],{"categories":573},[151],{"categories":575},[97],{"categories":577},[156],{"categories":579},[],{"categories":581},[170],{"categories":583},[156],{"categories":585},[97],{"categories":587},[159],{"categories":589},[97],{"categories":591},[592],"AI Policy & Regulation",{"categories":594},[],{"categories":596},[97],{"categories":598},[156],{"categories":600},[159],{"categories":602},[156],{"categories":604},[97],{"categories":606},[97],{"categories":608},[97],{"categories":610},[156],{"categories":612},[],{"categories":614},[195],{"categories":616},[617],"Evals & Reliability",{"categories":619},[97],{"categories":621},[97],{"categories":623},[],{"categories":625},[146],{"categories":627},[408],{"categories":629},[592],{"categories":631},[97],{"categories":633},[151],{"categories":635},[97],{"categories":637},[156],{"categories":639},[97],{"categories":641},[156],{"categories":643},[487],{"categories":645},[97],{"categories":647},[170],{"categories":649},[97],{"categories":651},[],{"categories":653},[244],{"categories":655},[],{"categories":657},[97],{"categories":659},[408],{"categories":661},[97],{"categories":663},[97],{"categories":665},[97],{"categories":667},[],{"categories":669},[97],{"categories":671},[244],{"categories":673},[170],{"categories":675},[],{"categories":677},[97],{"categories":679},[],{"categories":681},[156],{"categories":683},[97],{"categories":685},[244],{"categories":687},[],{"categories":689},[97],{"categories":691},[97],{"categories":693},[195],{"categories":695},[156],{"categories":697},[97],{"categories":699},[151],{"categories":701},[156],{"categories":703},[97],{"categories":705},[97],{"categories":707},[170],{"categories":709},[244],{"categories":711},[97],{"categories":713},[156],{"categories":715},[],{"categories":717},[170],{"categories":719},[156],{"categories":721},[195],{"categories":723},[],{"categories":725},[97],{"categories":727},[192],{"categories":729},[97],{"categories":731},[],{"categories":733},[97],{"categories":735},[97],{"categories":737},[97],{"categories":739},[151,273],{"categories":741},[],{"categories":743},[170],{"categories":745},[97],{"categories":747},[97],{"categories":749},[156],{"categories":751},[97],{"categories":753},[],{"categories":755},[],{"categories":757},[97],{"categories":759},[244],{"categories":761},[97],{"categories":763},[],{"categories":765},[97],{"categories":767},[314],{"categories":769},[],{"categories":771},[156],{"categories":773},[192],{"categories":775},[97],{"categories":777},[97],{"categories":779},[244],{"categories":781},[],{"categories":783},[192],{"categories":785},[97],{"categories":787},[181],{"categories":789},[97],{"categories":791},[97],{"categories":793},[156],{"categories":795},[192],{"categories":797},[546],{"categories":799},[97],{"categories":801},[273],{"categories":803},[],{"categories":805},[156],{"categories":807},[151],{"categories":809},[170],{"categories":811},[97],{"categories":813},[156],{"categories":815},[],{"categories":817},[97,314],{"categories":819},[97],{"categories":821},[97],{"categories":823},[97],{"categories":825},[156],{"categories":827},[97,170],{"categories":829},[195],{"categories":831},[97],{"categories":833},[97],{"categories":835},[97],{"categories":837},[170],{"categories":839},[97],{"categories":841},[156],{"categories":843},[156],{"categories":845},[592],{"categories":847},[273],{"categories":849},[97],{"categories":851},[156],{"categories":853},[97],{"categories":855},[97],{"categories":857},[156],{"categories":859},[],{"categories":861},[156],{"categories":863},[97],{"categories":865},[97],{"categories":867},[156],{"categories":869},[97],{"categories":871},[97,151],{"categories":873},[97],{"categories":875},[151],{"categories":877},[],{"categories":879},[244],{"categories":881},[244],{"categories":883},[97],{"categories":885},[],{"categories":887},[],{"categories":889},[97],{"categories":891},[192],{"categories":893},[],{"categories":895},[146],{"categories":897},[97],{"categories":899},[170],{"categories":901},[97],{"categories":903},[904],"Generative UI & Design-to-Code",{"categories":906},[97],{"categories":908},[97],{"categories":910},[244],{"categories":912},[97],{"categories":914},[915],"Algorithmic Accountability",{"categories":917},[156],{"categories":919},[170],{"categories":921},[192],{"categories":923},[244],{"categories":925},[97],{"categories":927},[],{"categories":929},[159],{"categories":931},[97],{"categories":933},[97],{"categories":935},[97],{"categories":937},[97],{"categories":939},[156],{"categories":941},[942],"MLOps & Infrastructure",{"categories":944},[97],{"categories":946},[97],{"categories":948},[97],{"categories":950},[97],{"categories":952},[97],{"categories":954},[170],{"categories":956},[192],{"categories":958},[97],{"categories":960},[159],{"categories":962},[146],{"categories":964},[97],{"categories":966},[156],{"categories":968},[314],{"categories":970},[97],{"categories":972},[151],{"categories":974},[97],{"categories":976},[244],{"categories":978},[97],{"categories":980},[97],{"categories":982},[156],{"categories":984},[],{"categories":986},[],{"categories":988},[97],{"categories":990},[181],{"categories":992},[244],{"categories":994},[192],{"categories":996},[195],{"categories":998},[],{"categories":1000},[97],{"categories":1002},[97],{"categories":1004},[151],{"categories":1006},[156],{"categories":1008},[97],{"categories":1010},[97],{"categories":1012},[97],{"categories":1014},[97],{"categories":1016},[192],{"categories":1018},[181],{"categories":1020},[97],{"categories":1022},[244],{"categories":1024},[97],{"categories":1026},[],{"categories":1028},[156],{"categories":1030},[170],{"categories":1032},[],{"categories":1034},[97],{"categories":1036},[97],{"categories":1038},[156],{"categories":1040},[170],{"categories":1042},[97],{"categories":1044},[195],{"categories":1046},[244],{"categories":1048},[],{"categories":1050},[97],{"categories":1052},[],{"categories":1054},[97],{"categories":1056},[],{"categories":1058},[97],{"categories":1060},[97],{"categories":1062},[159],{"categories":1064},[151],{"categories":1066},[156],{"categories":1068},[156],{"categories":1070},[],{"categories":1072},[97],{"categories":1074},[146],{"categories":1076},[97],{"categories":1078},[97],{"categories":1080},[151],{"categories":1082},[192],{"categories":1084},[146],{"categories":1086},[],{"categories":1088},[97],{"categories":1090},[],{"categories":1092},[97],{"categories":1094},[],{"categories":1096},[192],{"categories":1098},[192],{"categories":1100},[],{"categories":1102},[487],{"categories":1104},[97],{"categories":1106},[244],{"categories":1108},[170],{"categories":1110},[],{"categories":1112},[431],{"categories":1114},[156],{"categories":1116},[151],{"categories":1118},[],{"categories":1120},[],{"categories":1122},[146],{"categories":1124},[195],{"categories":1126},[],{"categories":1128},[273],{"categories":1130},[156],{"categories":1132},[151],{"categories":1134},[156],{"categories":1136},[97],{"categories":1138},[151],{"categories":1140},[97],{"categories":1142},[170],{"categories":1144},[],{"categories":1146},[181],{"categories":1148},[159],{"categories":1150},[97],{"categories":1152},[244],{"categories":1154},[170],{"categories":1156},[151],{"categories":1158},[97],{"categories":1160},[170],{"categories":1162},[97],{"categories":1164},[156],{"categories":1166},[151],{"categories":1168},[97],{"categories":1170},[97],{"categories":1172},[97],{"categories":1174},[97],{"categories":1176},[97],{"categories":1178},[],{"categories":1180},[],{"categories":1182},[170],{"categories":1184},[195],{"categories":1186},[159],{"categories":1188},[97],{"categories":1190},[156],{"categories":1192},[170],{"categories":1194},[170],{"categories":1196},[97],{"categories":1198},[],{"categories":1200},[192],{"categories":1202},[159],{"categories":1204},[159],{"categories":1206},[170],{"categories":1208},[97],{"categories":1210},[617],{"categories":1212},[314],{"categories":1214},[],{"categories":1216},[156],{"categories":1218},[97],{"categories":1220},[],{"categories":1222},[146],{"categories":1224},[],{"categories":1226},[97],{"categories":1228},[97],{"categories":1230},[97],{"categories":1232},[244],{"categories":1234},[273],{"categories":1236},[97],{"categories":1238},[170],{"categories":1240},[97],{"categories":1242},[156],{"categories":1244},[],{"categories":1246},[170],{"categories":1248},[97],{"categories":1250},[146],{"categories":1252},[],{"categories":1254},[151],{"categories":1256},[97],{"categories":1258},[97],{"categories":1260},[192],{"categories":1262},[97,314],{"categories":1264},[97],{"categories":1266},[1267],"Design Systems for AI",{"categories":1269},[97],{"categories":1271},[97],{"categories":1273},[192],{"categories":1275},[97],{"categories":1277},[97],{"categories":1279},[97],{"categories":1281},[151],{"categories":1283},[97],{"categories":1285},[97],{"categories":1287},[97],{"categories":1289},[],{"categories":1291},[97],{"categories":1293},[97],{"categories":1295},[151],{"categories":1297},[97],{"categories":1299},[],{"categories":1301},[156],{"categories":1303},[156],{"categories":1305},[170],{"categories":1307},[192],{"categories":1309},[170],{"categories":1311},[97],{"categories":1313},[244],{"categories":1315},[192],{"categories":1317},[195],{"categories":1319},[97],{"categories":1321},[97],{"categories":1323},[156],{"categories":1325},[146],{"categories":1327},[592],{"categories":1329},[97],{"categories":1331},[156],{"categories":1333},[97],{"categories":1335},[170],{"categories":1337},[170],{"categories":1339},[],{"categories":1341},[],{"categories":1343},[97],{"categories":1345},[156],{"categories":1347},[159],{"categories":1349},[],{"categories":1351},[151],{"categories":1353},[97],{"categories":1355},[],{"categories":1357},[244],{"categories":1359},[170],{"categories":1361},[156],{"categories":1363},[170],{"categories":1365},[244],{"categories":1367},[97],{"categories":1369},[97],{"categories":1371},[244],{"categories":1373},[],{"categories":1375},[],{"categories":1377},[192],{"categories":1379},[156],{"categories":1381},[156],{"categories":1383},[97],{"categories":1385},[97],{"categories":1387},[97],{"categories":1389},[97],{"categories":1391},[151],{"categories":1393},[97],{"categories":1395},[97],{"categories":1397},[],{"categories":1399},[170],{"categories":1401},[170],{"categories":1403},[97],{"categories":1405},[170],{"categories":1407},[151],{"categories":1409},[],{"categories":1411},[97],{"categories":1413},[97],{"categories":1415},[97],{"categories":1417},[97],{"categories":1419},[97],{"categories":1421},[156],{"categories":1423},[146],{"categories":1425},[151],{"categories":1427},[97],{"categories":1429},[156],{"categories":1431},[192],{"categories":1433},[156],{"categories":1435},[181],{"categories":1437},[273],{"categories":1439},[97],{"categories":1441},[156],{"categories":1443},[97],{"categories":1445},[97],{"categories":1447},[97],{"categories":1449},[],{"categories":1451},[244],{"categories":1453},[],{"categories":1455},[97],{"categories":1457},[97],{"categories":1459},[],{"categories":1461},[97],{"categories":1463},[170],{"categories":1465},[151],{"categories":1467},[1468],"Visual & Generative Media",{"categories":1470},[156],{"categories":1472},[],{"categories":1474},[97],{"categories":1476},[97],{"categories":1478},[170],{"categories":1480},[314],{"categories":1482},[97],{"categories":1484},[195],{"categories":1486},[592],{"categories":1488},[170],{"categories":1490},[273],{"categories":1492},[97],{"categories":1494},[244],{"categories":1496},[97],{"categories":1498},[97],{"categories":1500},[170],{"categories":1502},[156],{"categories":1504},[97],{"categories":1506},[],{"categories":1508},[],{"categories":1510},[156],{"categories":1512},[170],{"categories":1514},[146],{"categories":1516},[156],{"categories":1518},[546],{"categories":1520},[97],{"categories":1522},[159],{"categories":1524},[97],{"categories":1526},[151],{"categories":1528},[],{"categories":1530},[97],{"categories":1532},[159],{"categories":1534},[97],{"categories":1536},[97],{"categories":1538},[97],{"categories":1540},[159],{"categories":1542},[97],{"categories":1544},[97],{"categories":1546},[273],{"categories":1548},[97],{"categories":1550},[487],{"categories":1552},[97],{"categories":1554},[156],{"categories":1556},[97],{"categories":1558},[97],{"categories":1560},[97],{"categories":1562},[97],{"categories":1564},[244],{"categories":1566},[156],{"categories":1568},[],{"categories":1570},[156],{"categories":1572},[],{"categories":1574},[314],{"categories":1576},[170],{"categories":1578},[],{"categories":1580},[546],{"categories":1582},[97],{"categories":1584},[156],{"categories":1586},[156],{"categories":1588},[97],{"categories":1590},[244,97],{"categories":1592},[146],{"categories":1594},[97],{"categories":1596},[244],{"categories":1598},[],{"categories":1600},[97],{"categories":1602},[146],{"categories":1604},[97],{"categories":1606},[1607],"Medical Imaging & Radiology",{"categories":1609},[97],{"categories":1611},[97],{"categories":1613},[97],{"categories":1615},[244],{"categories":1617},[156],{"categories":1619},[170],{"categories":1621},[],{"categories":1623},[97],{"categories":1625},[97],{"categories":1627},[97],{"categories":1629},[],{"categories":1631},[],{"categories":1633},[97],{"categories":1635},[97],{"categories":1637},[487],{"categories":1639},[97],{"categories":1641},[146],{"categories":1643},[97],{"categories":1645},[97],{"categories":1647},[],{"categories":1649},[156],{"categories":1651},[97],{"categories":1653},[159],{"categories":1655},[170],{"categories":1657},[97],{"categories":1659},[156],{"categories":1661},[487],{"categories":1663},[97],{"categories":1665},[156],{"categories":1667},[97],{"categories":1669},[97],{"categories":1671},[97],{"categories":1673},[244],{"categories":1675},[156],{"categories":1677},[314],{"categories":1679},[244],{"categories":1681},[151],{"categories":1683},[156],{"categories":1685},[192],{"categories":1687},[97],{"categories":1689},[97],{"categories":1691},[159],{"categories":1693},[97],{"categories":1695},[97],{"categories":1697},[97],{"categories":1699},[97],{"categories":1701},[156],{"categories":1703},[97],{"categories":1705},[170],{"categories":1707},[170],{"categories":1709},[97],{"categories":1711},[159],{"categories":1713},[],{"categories":1715},[192],{"categories":1717},[],{"categories":1719},[159],{"categories":1721},[156],{"categories":1723},[97],{"categories":1725},[156],{"categories":1727},[1267],{"categories":1729},[1267],{"categories":1731},[244],{"categories":1733},[97],{"categories":1735},[97],{"categories":1737},[97],{"categories":1739},[156],{"categories":1741},[170],{"categories":1743},[244],{"categories":1745},[156],{"categories":1747},[192],{"categories":1749},[],{"categories":1751},[97],{"categories":1753},[],{"categories":1755},[97],{"categories":1757},[97],{"categories":1759},[97],{"categories":1761},[97],{"categories":1763},[156],{"categories":1765},[1766],"Contract Review & E-Discovery",{"categories":1768},[97],{"categories":1770},[244],{"categories":1772},[97],{"categories":1774},[146],{"categories":1776},[97],{"categories":1778},[192],{"categories":1780},[97],{"categories":1782},[97],{"categories":1784},[273],{"categories":1786},[170],{"categories":1788},[97],{"categories":1790},[97],{"categories":1792},[156],{"categories":1794},[156],{"categories":1796},[915],{"categories":1798},[97],{"categories":1800},[97],{"categories":1802},[156],{"categories":1804},[156],{"categories":1806},[97],{"categories":1808},[97],{"categories":1810},[97],{"categories":1812},[156],{"categories":1814},[97],{"categories":1816},[97],{"categories":1818},[487],{"categories":1820},[464],{"categories":1822},[97],{"categories":1824},[156],{"categories":1826},[97],{"categories":1828},[1829],"Law-Firm Practice & Adoption",{"categories":1831},[97],{"categories":1833},[156],{"categories":1835},[244],{"categories":1837},[97],{"categories":1839},[97],{"categories":1841},[97],{"categories":1843},[],{"categories":1845},[],{"categories":1847},[170],{"categories":1849},[97],{"categories":1851},[],{"categories":1853},[156],{"categories":1855},[146],{"categories":1857},[314],{"categories":1859},[97],{"categories":1861},[],{"categories":1863},[146],{"categories":1865},[151],{"categories":1867},[97],{"categories":1869},[273],{"categories":1871},[],{"categories":1873},[151],{"categories":1875},[156],{"categories":1877},[151],{"categories":1879},[],{"categories":1881},[97],{"categories":1883},[159],{"categories":1885},[97],{"categories":1887},[170],{"categories":1889},[],{"categories":1891},[],{"categories":1893},[],{"categories":1895},[],{"categories":1897},[97],{"categories":1899},[159],{"categories":1901},[156],{"categories":1903},[314],{"categories":1905},[97],{"categories":1907},[146],{"categories":1909},[170],{"categories":1911},[97],{"categories":1913},[97],{"categories":1915},[170],{"categories":1917},[159],{"categories":1919},[97],{"categories":1921},[97],{"categories":1923},[97],{"categories":1925},[942],{"categories":1927},[97],{"categories":1929},[170],{"categories":1931},[97],{"categories":1933},[273],{"categories":1935},[170],{"categories":1937},[151],{"categories":1939},[97],{"categories":1941},[97],{"categories":1943},[97],{"categories":1945},[244],{"categories":1947},[97],{"categories":1949},[97],{"categories":1951},[97],{"categories":1953},[97],{"categories":1955},[151],{"categories":1957},[156],{"categories":1959},[97,146],{"categories":1961},[487],{"categories":1963},[97],{"categories":1965},[97],{"categories":1967},[170],{"categories":1969},[170],{"categories":1971},[244],{"categories":1973},[156],{"categories":1975},[156],{"categories":1977},[170],{"categories":1979},[97],{"categories":1981},[97],{"categories":1983},[97],{"categories":1985},[],{"categories":1987},[],{"categories":1989},[97],{"categories":1991},[195],{"categories":1993},[97],{"categories":1995},[156],{"categories":1997},[],{"categories":1999},[97],{"categories":2001},[97],{"categories":2003},[170],{"categories":2005},[195],{"categories":2007},[192],{"categories":2009},[244],{"categories":2011},[97],{"categories":2013},[156],{"categories":2015},[97],{"categories":2017},[170],{"categories":2019},[],{"categories":2021},[156],{"categories":2023},[97],{"categories":2025},[97],{"categories":2027},[97],{"categories":2029},[97],{"categories":2031},[],{"categories":2033},[156],{"categories":2035},[97],{"categories":2037},[97],{"categories":2039},[97],{"categories":2041},[],{"categories":2043},[156],{"categories":2045},[97],{"categories":2047},[97],{"categories":2049},[151],{"categories":2051},[97],{"categories":2053},[97],{"categories":2055},[],{"categories":2057},[146],{"categories":2059},[97],{"categories":2061},[97],{"categories":2063},[97],{"categories":2065},[244],{"categories":2067},[97],{"categories":2069},[170],{"categories":2071},[97],{"categories":2073},[146],{"categories":2075},[97],{"categories":2077},[170],{"categories":2079},[273],{"categories":2081},[156],{"categories":2083},[156],{"categories":2085},[97],{"categories":2087},[97],{"categories":2089},[97,244],{"categories":2091},[97],{"categories":2093},[156],{"categories":2095},[192],{"categories":2097},[97],{"categories":2099},[192],{"categories":2101},[156],{"categories":2103},[244],{"categories":2105},[97],{"categories":2107},[],{"categories":2109},[170],{"categories":2111},[314],{"categories":2113},[244],{"categories":2115},[170],{"categories":2117},[97],{"categories":2119},[159],{"categories":2121},[97],{"categories":2123},[97],{"categories":2125},[156],{"categories":2127},[],{"categories":2129},[],{"categories":2131},[97],{"categories":2133},[],{"categories":2135},[],{"categories":2137},[159],{"categories":2139},[170],{"categories":2141},[97],{"categories":2143},[156],{"categories":2145},[156],{"categories":2147},[151],{"categories":2149},[156],{"categories":2151},[314],{"categories":2153},[97],{"categories":2155},[97],{"categories":2157},[97],{"categories":2159},[181],{"categories":2161},[97],{"categories":2163},[97],{"categories":2165},[97],{"categories":2167},[170],{"categories":2169},[156],{"categories":2171},[97],{"categories":2173},[97],{"categories":2175},[170],{"categories":2177},[431],{"categories":2179},[156],{"categories":2181},[915],{"categories":2183},[],{"categories":2185},[244],{"categories":2187},[1829],{"categories":2189},[170],{"categories":2191},[],{"categories":2193},[],{"categories":2195},[97],{"categories":2197},[156],{"categories":2199},[],{"categories":2201},[],{"categories":2203},[97],{"categories":2205},[273],{"categories":2207},[97],{"categories":2209},[273],{"categories":2211},[156],{"categories":2213},[97],{"categories":2215},[97],{"categories":2217},[170],{"categories":2219},[159],{"categories":2221},[],{"categories":2223},[97],{"categories":2225},[97],{"categories":2227},[170],{"categories":2229},[1766],{"categories":2231},[244],{"categories":2233},[244],{"categories":2235},[97],{"categories":2237},[156],{"categories":2239},[146],{"categories":2241},[97],{"categories":2243},[97],{"categories":2245},[97],{"categories":2247},[97],{"categories":2249},[244],{"categories":2251},[244],{"categories":2253},[156],{"categories":2255},[156],{"categories":2257},[156],{"categories":2259},[97],{"categories":2261},[97],{"categories":2263},[],{"categories":2265},[97],{"categories":2267},[],{"categories":2269},[2270],"Interaction & Product Design",{"categories":2272},[97],{"categories":2274},[156],{"categories":2276},[170],{"categories":2278},[343],{"categories":2280},[192],{"categories":2282},[170],{"categories":2284},[97],{"categories":2286},[97],{"categories":2288},[97],{"categories":2290},[170],{"categories":2292},[97],{"categories":2294},[146],{"categories":2296},[156],{"categories":2298},[97],{"categories":2300},[],{"categories":2302},[156],{"categories":2304},[156],{"categories":2306},[156],{"categories":2308},[],{"categories":2310},[170],{"categories":2312},[97],{"categories":2314},[156],{"categories":2316},[146],{"categories":2318},[2270],{"categories":2320},[97],{"categories":2322},[146],{"categories":2324},[146],{"categories":2326},[],{"categories":2328},[156],{"categories":2330},[170],{"categories":2332},[],{"categories":2334},[156],{"categories":2336},[192],{"categories":2338},[97],{"categories":2340},[156],{"categories":2342},[97],{"categories":2344},[156],{"categories":2346},[156],{"categories":2348},[97],{"categories":2350},[97],{"categories":2352},[192],{"categories":2354},[195],{"categories":2356},[97],{"categories":2358},[159],{"categories":2360},[170],{"categories":2362},[2363],"Coding Agents & Dev Productivity",{"categories":2365},[192],{"categories":2367},[244],{"categories":2369},[97],{"categories":2371},[97],{"categories":2373},[],{"categories":2375},[97],{"categories":2377},[915],{"categories":2379},[],{"categories":2381},[97],{"categories":2383},[97],{"categories":2385},[314],{"categories":2387},[97],{"categories":2389},[192],{"categories":2391},[],{"categories":2393},[],{"categories":2395},[97],{"categories":2397},[],{"categories":2399},[156],{"categories":2401},[97],{"categories":2403},[],{"categories":2405},[170],{"categories":2407},[170],{"categories":2409},[97],{"categories":2411},[195],{"categories":2413},[],{"categories":2415},[97],{"categories":2417},[97],{"categories":2419},[97],{"categories":2421},[195],{"categories":2423},[170],{"categories":2425},[156],{"categories":2427},[],{"categories":2429},[],{"categories":2431},[97],{"categories":2433},[97],{"categories":2435},[156],{"categories":2437},[156],{"categories":2439},[408],{"categories":2441},[170],{"categories":2443},[170],{"categories":2445},[156],{"categories":2447},[192],{"categories":2449},[192],{"categories":2451},[156],{"categories":2453},[156],{"categories":2455},[97],{"categories":2457},[146],{"categories":2459},[2270],{"categories":2461},[159],{"categories":2463},[97,314],{"categories":2465},[195],{"categories":2467},[],{"categories":2469},[244],{"categories":2471},[156],{"categories":2473},[170],{"categories":2475},[146],{"categories":2477},[97],{"categories":2479},[156],{"categories":2481},[2482],"The Designer's Role & Craft",{"categories":2484},[244],{"categories":2486},[],{"categories":2488},[156],{"categories":2490},[97],{"categories":2492},[156],{"categories":2494},[156],{"categories":2496},[97],{"categories":2498},[273],{"categories":2500},[97],{"categories":2502},[170],{"categories":2504},[97],{"categories":2506},[244],{"categories":2508},[97],{"categories":2510},[],{"categories":2512},[156],{"categories":2514},[244],{"categories":2516},[159],{"categories":2518},[97],{"categories":2520},[97],{"categories":2522},[97],{"categories":2524},[2525],"AI UX Patterns",{"categories":2527},[156],{"categories":2529},[156],{"categories":2531},[156],{"categories":2533},[156],{"categories":2535},[273],{"categories":2537},[195],{"categories":2539},[97],{"categories":2541},[156],{"categories":2543},[97],{"categories":2545},[1267],{"categories":2547},[],{"categories":2549},[273],{"categories":2551},[156],{"categories":2553},[192],{"categories":2555},[170],{"categories":2557},[97],{"categories":2559},[156],{"categories":2561},[],{"categories":2563},[],{"categories":2565},[97],{"categories":2567},[97],{"categories":2569},[156],{"categories":2571},[97],{"categories":2573},[156],{"categories":2575},[408],{"categories":2577},[244],{"categories":2579},[97],{"categories":2581},[192],{"categories":2583},[170],{"categories":2585},[97],{"categories":2587},[156],{"categories":2589},[156],{"categories":2591},[],{"categories":2593},[97],{"categories":2595},[],{"categories":2597},[97],{"categories":2599},[],{"categories":2601},[97],{"categories":2603},[97],{"categories":2605},[97],{"categories":2607},[156],{"categories":2609},[170],{"categories":2611},[],{"categories":2613},[],{"categories":2615},[195],{"categories":2617},[181],{"categories":2619},[97],{"categories":2621},[97],{"categories":2623},[97],{"categories":2625},[195],{"categories":2627},[97],{"categories":2629},[97],{"categories":2631},[192],{"categories":2633},[97],{"categories":2635},[97],{"categories":2637},[97],{"categories":2639},[156],{"categories":2641},[97],{"categories":2643},[156],{"categories":2645},[97],{"categories":2647},[97],{"categories":2649},[97],{"categories":2651},[156],{"categories":2653},[],{"categories":2655},[97],{"categories":2657},[],{"categories":2659},[97],{"categories":2661},[97],{"categories":2663},[314],{"categories":2665},[97],{"categories":2667},[],{"categories":2669},[],{"categories":2671},[244],{"categories":2673},[942],{"categories":2675},[156],{"categories":2677},[146],{"categories":2679},[2482],{"categories":2681},[],{"categories":2683},[],{"categories":2685},[97],{"categories":2687},[],{"categories":2689},[],{"categories":2691},[170],{"categories":2693},[192],{"categories":2695},[273],{"categories":2697},[156],{"categories":2699},[151],{"categories":2701},[97],{"categories":2703},[97],{"categories":2705},[151],{"categories":2707},[],{"categories":2709},[244],{"categories":2711},[159],{"categories":2713},[97],{"categories":2715},[97],{"categories":2717},[156],{"categories":2719},[151],{"categories":2721},[97],{"categories":2723},[97],{"categories":2725},[146],{"categories":2727},[97],{"categories":2729},[97],{"categories":2731},[],{"categories":2733},[146],{"categories":2735},[97],{"categories":2737},[273],{"categories":2739},[156],{"categories":2741},[192],{"categories":2743},[97],{"categories":2745},[170],{"categories":2747},[97],{"categories":2749},[97],{"categories":2751},[151],{"categories":2753},[97],{"categories":2755},[97],{"categories":2757},[97],{"categories":2759},[156],{"categories":2761},[97],{"categories":2763},[],{"categories":2765},[97],{"categories":2767},[170],{"categories":2769},[146],{"categories":2771},[97],{"categories":2773},[97],{"categories":2775},[97],{"categories":2777},[],{"categories":2779},[97],{"categories":2781},[487],{"categories":2783},[156],{"categories":2785},[151],{"categories":2787},[192],{"categories":2789},[97],{"categories":2791},[97],{"categories":2793},[],{"categories":2795},[151],{"categories":2797},[151],{"categories":2799},[97],{"categories":2801},[97],{"categories":2803},[159],{"categories":2805},[97],{"categories":2807},[97],{"categories":2809},[97],{"categories":2811},[97],{"categories":2813},[170],{"categories":2815},[170],{"categories":2817},[97],{"categories":2819},[],{"categories":2821},[170],{"categories":2823},[97],{"categories":2825},[170],{"categories":2827},[156],{"categories":2829},[592],{"categories":2831},[],{"categories":2833},[],{"categories":2835},[97],{"categories":2837},[192],{"categories":2839},[],{"categories":2841},[314],{"categories":2843},[97],{"categories":2845},[97],{"categories":2847},[97],{"categories":2849},[244],{"categories":2851},[904],{"categories":2853},[],{"categories":2855},[97],{"categories":2857},[97],{"categories":2859},[97],{"categories":2861},[170],{"categories":2863},[97],{"categories":2865},[97],{"categories":2867},[97,314],{"categories":2869},[97],{"categories":2871},[97],{"categories":2873},[244],{"categories":2875},[156],{"categories":2877},[],{"categories":2879},[156],{"categories":2881},[156],{"categories":2883},[97],{"categories":2885},[97],{"categories":2887},[97],{"categories":2889},[97],{"categories":2891},[195],{"categories":2893},[97],{"categories":2895},[2525],{"categories":2897},[146],{"categories":2899},[195],{"categories":2901},[146],{"categories":2903},[170],{"categories":2905},[244],{"categories":2907},[156],{"categories":2909},[97],{"categories":2911},[],{"categories":2913},[151],{"categories":2915},[97],{"categories":2917},[97],{"categories":2919},[192],{"categories":2921},[97],{"categories":2923},[97],{"categories":2925},[97],{"categories":2927},[156],{"categories":2929},[97],{"categories":2931},[97],{"categories":2933},[97],{"categories":2935},[151],{"categories":2937},[],{"categories":2939},[314],{"categories":2941},[97],{"categories":2943},[408],{"categories":2945},[244],{"categories":2947},[244],{"categories":2949},[170],{"categories":2951},[156],{"categories":2953},[97],{"categories":2955},[151],{"categories":2957},[192],{"categories":2959},[97],{"categories":2961},[97],{"categories":2963},[97],{"categories":2965},[244],{"categories":2967},[156],{"categories":2969},[156],{"categories":2971},[97],{"categories":2973},[97],{"categories":2975},[546],{"categories":2977},[156],{"categories":2979},[],{"categories":2981},[97],{"categories":2983},[97],{"categories":2985},[97],{"categories":2987},[],{"categories":2989},[],{"categories":2991},[97],{"categories":2993},[97],{"categories":2995},[156],{"categories":2997},[97],{"categories":2999},[97],{"categories":3001},[97],{"categories":3003},[170],{"categories":3005},[97],{"categories":3007},[97],{"categories":3009},[156],{"categories":3011},[97],{"categories":3013},[97],{"categories":3015},[97],{"categories":3017},[97],{"categories":3019},[97],{"categories":3021},[],{"categories":3023},[170],{"categories":3025},[195],{"categories":3027},[97],{"categories":3029},[156],{"categories":3031},[156],{"categories":3033},[97],{"categories":3035},[97],{"categories":3037},[],{"categories":3039},[],{"categories":3041},[97],{"categories":3043},[97],{"categories":3045},[97],{"categories":3047},[192],{"categories":3049},[195],{"categories":3051},[],{"categories":3053},[97],{"categories":3055},[244],{"categories":3057},[97],{"categories":3059},[314],{"categories":3061},[1829],{"categories":3063},[192],{"categories":3065},[170],{"categories":3067},[97],{"categories":3069},[170],{"categories":3071},[170],{"categories":3073},[97],{"categories":3075},[97],{"categories":3077},[170],{"categories":3079},[192],{"categories":3081},[192],{"categories":3083},[314],{"categories":3085},[156],{"categories":3087},[],{"categories":3089},[192],{"categories":3091},[97],{"categories":3093},[156],{"categories":3095},[146],{"categories":3097},[170],{"categories":3099},[97],{"categories":3101},[192],{"categories":3103},[],{"categories":3105},[97],{"categories":3107},[170],{"categories":3109},[170],{"categories":3111},[195],{"categories":3113},[97],{"categories":3115},[192],{"categories":3117},[97],{"categories":3119},[170],{"categories":3121},[156],{"categories":3123},[156],{"categories":3125},[192],{"categories":3127},[156],{"categories":3129},[314],{"categories":3131},[156],{"categories":3133},[97],{"categories":3135},[97],{"categories":3137},[97],{"categories":3139},[97],{"categories":3141},[170],{"categories":3143},[97],{"categories":3145},[],{"categories":3147},[156],{"categories":3149},[151],{"categories":3151},[170],{"categories":3153},[],{"categories":3155},[],{"categories":3157},[97],{"categories":3159},[156],{"categories":3161},[97],{"categories":3163},[97],{"categories":3165},[3166],"Frameworks & Tooling",{"categories":3168},[97],{"categories":3170},[97],{"categories":3172},[170],{"categories":3174},[97],{"categories":3176},[97],{"categories":3178},[],{"categories":3180},[195],{"categories":3182},[195],{"categories":3184},[146],{"categories":3186},[97],{"categories":3188},[156],{"categories":3190},[97],{"categories":3192},[244],{"categories":3194},[],{"categories":3196},[1829],{"categories":3198},[97],{"categories":3200},[170],{"categories":3202},[97],{"categories":3204},[314],{"categories":3206},[314],{"categories":3208},[],{"categories":3210},[156],{"categories":3212},[156],{"categories":3214},[97],{"categories":3216},[97],{"categories":3218},[192],{"categories":3220},[156],{"categories":3222},[192],{"categories":3224},[97],{"categories":3226},[156],{"categories":3228},[],{"categories":3230},[244],{"categories":3232},[97],{"categories":3234},[97],{"categories":3236},[],{"categories":3238},[97],{"categories":3240},[156],{"categories":3242},[97],{"categories":3244},[97],{"categories":3246},[97],{"categories":3248},[],{"categories":3250},[170],{"categories":3252},[97],{"categories":3254},[170],{"categories":3256},[314],{"categories":3258},[97],{"categories":3260},[97],{"categories":3262},[97],{"categories":3264},[170],{"categories":3266},[151],{"categories":3268},[97],{"categories":3270},[1829],{"categories":3272},[],{"categories":3274},[156],{"categories":3276},[146],{"categories":3278},[97],{"categories":3280},[146],{"categories":3282},[97],{"categories":3284},[],{"categories":3286},[156],{"categories":3288},[97],{"categories":3290},[97],{"categories":3292},[3293],"AI Design Tooling",{"categories":3295},[244],{"categories":3297},[97],{"categories":3299},[97],{"categories":3301},[170],{"categories":3303},[244],{"categories":3305},[97],{"categories":3307},[97],{"categories":3309},[170],{"categories":3311},[192],{"categories":3313},[159],{"categories":3315},[170],{"categories":3317},[97],{"categories":3319},[97],{"categories":3321},[97],{"categories":3323},[156],{"categories":3325},[97],{"categories":3327},[],{"categories":3329},[156],{"categories":3331},[97],{"categories":3333},[97],{"categories":3335},[156],{"categories":3337},[97],{"categories":3339},[97],{"categories":3341},[97],{"categories":3343},[156],{"categories":3345},[],{"categories":3347},[156],{"categories":3349},[3166],{"categories":3351},[97],{"categories":3353},[97],{"categories":3355},[156],{"categories":3357},[156],{"categories":3359},[170],{"categories":3361},[170],{"categories":3363},[97],{"categories":3365},[],{"categories":3367},[170],{"categories":3369},[97],{"categories":3371},[97],{"categories":3373},[156],{"categories":3375},[151],{"categories":3377},[97],{"categories":3379},[],{"categories":3381},[97],{"categories":3383},[97],{"categories":3385},[2270],{"categories":3387},[],{"categories":3389},[97],{"categories":3391},[97],{"categories":3393},[97],{"categories":3395},[97],{"categories":3397},[244],{"categories":3399},[97],{"categories":3401},[],{"categories":3403},[97],{"categories":3405},[97],{"categories":3407},[97],{"categories":3409},[97],{"categories":3411},[273],{"categories":3413},[192],{"categories":3415},[97],{"categories":3417},[97],{"categories":3419},[1829],{"categories":3421},[146],{"categories":3423},[97],{"categories":3425},[97],{"categories":3427},[195],{"categories":3429},[97],{"categories":3431},[97],{"categories":3433},[192],{"categories":3435},[156],{"categories":3437},[],{"categories":3439},[97],{"categories":3441},[97],{"categories":3443},[244],{"categories":3445},[97],{"categories":3447},[273],{"categories":3449},[156],{"categories":3451},[97],{"categories":3453},[156],{"categories":3455},[],{"categories":3457},[],{"categories":3459},[],{"categories":3461},[146],{"categories":3463},[192],{"categories":3465},[156],{"categories":3467},[97],{"categories":3469},[97],{"categories":3471},[97],{"categories":3473},[97],{"categories":3475},[431],{"categories":3477},[244],{"categories":3479},[156],{"categories":3481},[97],{"categories":3483},[],{"categories":3485},[156],{"categories":3487},[156],{"categories":3489},[],{"categories":3491},[97],{"categories":3493},[156],{"categories":3495},[97],{"categories":3497},[],{"categories":3499},[97],{"categories":3501},[97],{"categories":3503},[97],{"categories":3505},[192],{"categories":3507},[244],{"categories":3509},[156],{"categories":3511},[244],{"categories":3513},[156],{"categories":3515},[97],{"categories":3517},[151],{"categories":3519},[],{"categories":3521},[],{"categories":3523},[97],{"categories":3525},[97],{"categories":3527},[97],{"categories":3529},[146],{"categories":3531},[156],{"categories":3533},[192],{"categories":3535},[],{"categories":3537},[244],{"categories":3539},[],{"categories":3541},[170],{"categories":3543},[97],{"categories":3545},[170],{"categories":3547},[244],{"categories":3549},[170],{"categories":3551},[97],{"categories":3553},[],{"categories":3555},[97],{"categories":3557},[97],{"categories":3559},[],{"categories":3561},[97],{"categories":3563},[97],{"categories":3565},[273],{"categories":3567},[97],{"categories":3569},[97],{"categories":3571},[314],{"categories":3573},[170],{"categories":3575},[97],{"categories":3577},[],{"categories":3579},[156],{"categories":3581},[97],{"categories":3583},[146],{"categories":3585},[546],{"categories":3587},[97],{"categories":3589},[97],{"categories":3591},[156],{"categories":3593},[97],{"categories":3595},[156],{"categories":3597},[97],{"categories":3599},[97],{"categories":3601},[97],{"categories":3603},[97],{"categories":3605},[],{"categories":3607},[97],{"categories":3609},[146],{"categories":3611},[97],{"categories":3613},[151],{"categories":3615},[170],{"categories":3617},[244],{"categories":3619},[],{"categories":3621},[97],{"categories":3623},[],{"categories":3625},[156],{"categories":3627},[97],{"categories":3629},[],{"categories":3631},[156],{"categories":3633},[97],{"categories":3635},[170],{"categories":3637},[244],{"categories":3639},[192],{"categories":3641},[97],{"categories":3643},[192],{"categories":3645},[156],{"categories":3647},[244],{"categories":3649},[97],{"categories":3651},[],{"categories":3653},[97],{"categories":3655},[181],{"categories":3657},[156],{"categories":3659},[97],{"categories":3661},[244],{"categories":3663},[192],{"categories":3665},[151],{"categories":3667},[170],{"categories":3669},[97],{"categories":3671},[97],{"categories":3673},[97],{"categories":3675},[97],{"categories":3677},[192],{"categories":3679},[273],{"categories":3681},[],{"categories":3683},[],{"categories":3685},[195],{"categories":3687},[487],{"categories":3689},[97],{"categories":3691},[156],{"categories":3693},[97,170],{"categories":3695},[192],{"categories":3697},[97],{"categories":3699},[97],{"categories":3701},[97],{"categories":3703},[97],{"categories":3705},[97],{"categories":3707},[97],{"categories":3709},[97],{"categories":3711},[156],{"categories":3713},[97],{"categories":3715},[156],{"categories":3717},[97],{"categories":3719},[97],{"categories":3721},[97],{"categories":3723},[],{"categories":3725},[97],{"categories":3727},[1267],{"categories":3729},[170],{"categories":3731},[244],{"categories":3733},[97],{"categories":3735},[97],{"categories":3737},[97],{"categories":3739},[195],{"categories":3741},[156],{"categories":3743},[273],{"categories":3745},[314],{"categories":3747},[],{"categories":3749},[170],{"categories":3751},[97],{"categories":3753},[151],{"categories":3755},[156],{"categories":3757},[146],{"categories":3759},[156],{"categories":3761},[97],{"categories":3763},[156],{"categories":3765},[156],{"categories":3767},[159],{"categories":3769},[170],{"categories":3771},[97],{"categories":3773},[97],{"categories":3775},[],{"categories":3777},[],{"categories":3779},[],{"categories":3781},[314],{"categories":3783},[97],{"categories":3785},[192],{"categories":3787},[97],{"categories":3789},[97],{"categories":3791},[97],{"categories":3793},[97],{"categories":3795},[],{"categories":3797},[97],{"categories":3799},[195],{"categories":3801},[151],{"categories":3803},[156],{"categories":3805},[97],{"categories":3807},[],{"categories":3809},[97],{"categories":3811},[156],{"categories":3813},[97],{"categories":3815},[314],{"categories":3817},[],{"categories":3819},[244],{"categories":3821},[244],{"categories":3823},[97],{"categories":3825},[156],{"categories":3827},[],{"categories":3829},[170],{"categories":3831},[97],{"categories":3833},[244],{"categories":3835},[97],{"categories":3837},[151],{"categories":3839},[156],{"categories":3841},[97],{"categories":3843},[],{"categories":3845},[192],{"categories":3847},[97],{"categories":3849},[97],{"categories":3851},[97],{"categories":3853},[244],{"categories":3855},[156],{"categories":3857},[192],{"categories":3859},[],{"categories":3861},[156],{"categories":3863},[151],{"categories":3865},[156],{"categories":3867},[244],{"categories":3869},[97],{"categories":3871},[97],{"categories":3873},[97],{"categories":3875},[487],{"categories":3877},[97],{"categories":3879},[156],{"categories":3881},[],{"categories":3883},[97],{"categories":3885},[97],{"categories":3887},[314],{"categories":3889},[192],{"categories":3891},[195],{"categories":3893},[592],{"categories":3895},[195],{"categories":3897},[195],{"categories":3899},[97],{"categories":3901},[],{"categories":3903},[],{"categories":3905},[],{"categories":3907},[156],{"categories":3909},[97],{"categories":3911},[156],{"categories":3913},[156],{"categories":3915},[170],{"categories":3917},[97],{"categories":3919},[464],{"categories":3921},[170],{"categories":3923},[156],{"categories":3925},[97],{"categories":3927},[97],{"categories":3929},[97],{"categories":3931},[97],{"categories":3933},[97],{"categories":3935},[156],{"categories":3937},[97],{"categories":3939},[],{"categories":3941},[],{"categories":3943},[97],{"categories":3945},[],{"categories":3947},[97],{"categories":3949},[156],{"categories":3951},[244],{"categories":3953},[97],{"categories":3955},[97],{"categories":3957},[],{"categories":3959},[156],{"categories":3961},[97],{"categories":3963},[97],{"categories":3965},[159],{"categories":3967},[97],{"categories":3969},[244],{"categories":3971},[97],{"categories":3973},[156],{"categories":3975},[151],{"categories":3977},[97],{"categories":3979},[97],{"categories":3981},[273],{"categories":3983},[156],{"categories":3985},[97],{"categories":3987},[97],{"categories":3989},[904],{"categories":3991},[97],{"categories":3993},[156],{"categories":3995},[97],{"categories":3997},[170],{"categories":3999},[97],{"categories":4001},[546],{"categories":4003},[244],{"categories":4005},[],{"categories":4007},[97],{"categories":4009},[97],{"categories":4011},[192],{"categories":4013},[487],{"categories":4015},[156],{"categories":4017},[97],{"categories":4019},[],{"categories":4021},[192],{"categories":4023},[408],{"categories":4025},[156],{"categories":4027},[156],{"categories":4029},[156],{"categories":4031},[97],{"categories":4033},[97],{"categories":4035},[156],{"categories":4037},[],{"categories":4039},[151],{"categories":4041},[97],{"categories":4043},[151],{"categories":4045},[156],{"categories":4047},[],{"categories":4049},[170],{"categories":4051},[97],{"categories":4053},[97],{"categories":4055},[146],{"categories":4057},[97],{"categories":4059},[192],{"categories":4061},[314],{"categories":4063},[181],{"categories":4065},[156],{"categories":4067},[156],{"categories":4069},[97],{"categories":4071},[97],{"categories":4073},[156],{"categories":4075},[97],{"categories":4077},[146],{"categories":4079},[],{"categories":4081},[156],{"categories":4083},[97],{"categories":4085},[97],{"categories":4087},[97],{"categories":4089},[156],{"categories":4091},[97],{"categories":4093},[],{"categories":4095},[97],{"categories":4097},[],{"categories":4099},[244],{"categories":4101},[156],{"categories":4103},[97,151],{"categories":4105},[156],{"categories":4107},[97],{"categories":4109},[],{"categories":4111},[146],{"categories":4113},[195],{"categories":4115},[151],{"categories":4117},[97],{"categories":4119},[170],{"categories":4121},[97],{"categories":4123},[97],{"categories":4125},[156],{"categories":4127},[97],{"categories":4129},[97],{"categories":4131},[97],{"categories":4133},[192],{"categories":4135},[1267],{"categories":4137},[156],{"categories":4139},[97],{"categories":4141},[],{"categories":4143},[],{"categories":4145},[97],{"categories":4147},[156],{"categories":4149},[97],{"categories":4151},[97],{"categories":4153},[314],{"categories":4155},[],{"categories":4157},[97],{"categories":4159},[156],{"categories":4161},[181],{"categories":4163},[156],{"categories":4165},[487],{"categories":4167},[],{"categories":4169},[431],{"categories":4171},[156],{"categories":4173},[97],{"categories":4175},[97],{"categories":4177},[273],{"categories":4179},[156],{"categories":4181},[97],{"categories":4183},[195],{"categories":4185},[159],{"categories":4187},[156],{"categories":4189},[97],{"categories":4191},[487],{"categories":4193},[97],{"categories":4195},[314],{"categories":4197},[151],{"categories":4199},[],{"categories":4201},[97],{"categories":4203},[97],{"categories":4205},[273],{"categories":4207},[244],{"categories":4209},[97],{"categories":4211},[97],{"categories":4213},[97],{"categories":4215},[],{"categories":4217},[273],{"categories":4219},[192],{"categories":4221},[97],{"categories":4223},[97],{"categories":4225},[97],{"categories":4227},[592],{"categories":4229},[146],{"categories":4231},[97],{"categories":4233},[159],{"categories":4235},[97],{"categories":4237},[],{"categories":4239},[],{"categories":4241},[244],{"categories":4243},[97],{"categories":4245},[195],{"categories":4247},[273],{"categories":4249},[156],{"categories":4251},[97],{"categories":4253},[97],{"categories":4255},[273],{"categories":4257},[192],{"categories":4259},[97],{"categories":4261},[],{"categories":4263},[97],{"categories":4265},[97],{"categories":4267},[],{"categories":4269},[97],{"categories":4271},[97],{"categories":4273},[617],{"categories":4275},[97],{"categories":4277},[97],{"categories":4279},[156],{"categories":4281},[170],{"categories":4283},[487],{"categories":4285},[97],{"categories":4287},[97],{"categories":4289},[97],{"categories":4291},[],{"categories":4293},[97,170],{"categories":4295},[192],{"categories":4297},[156],{"categories":4299},[170],{"categories":4301},[156],{"categories":4303},[942],{"categories":4305},[170],{"categories":4307},[170],{"categories":4309},[156],{"categories":4311},[97],{"categories":4313},[146],{"categories":4315},[],{"categories":4317},[],{"categories":4319},[156],{"categories":4321},[97],{"categories":4323},[170],{"categories":4325},[97],{"categories":4327},[146],{"categories":4329},[170],{"categories":4331},[170],{"categories":4333},[97],{"categories":4335},[273],{"categories":4337},[97],{"categories":4339},[170],{"categories":4341},[97],{"categories":4343},[],{"categories":4345},[97],{"categories":4347},[97],{"categories":4349},[244,97],{"categories":4351},[314],{"categories":4353},[146],{"categories":4355},[97],{"categories":4357},[],{"categories":4359},[97],{"categories":4361},[97],{"categories":4363},[151],{"categories":4365},[97],{"categories":4367},[151],{"categories":4369},[97],{"categories":4371},[97],{"categories":4373},[408],{"categories":4375},[97],{"categories":4377},[151],{"categories":4379},[170],{"categories":4381},[195],{"categories":4383},[156],{"categories":4385},[97],{"categories":4387},[170],{"categories":4389},[97],{"categories":4391},[97],{"categories":4393},[192],{"categories":4395},[273],{"categories":4397},[244],{"categories":4399},[97],{"categories":4401},[97],{"categories":4403},[97],{"categories":4405},[97],{"categories":4407},[146],{"categories":4409},[97],{"categories":4411},[156],{"categories":4413},[156],{"categories":4415},[170],{"categories":4417},[192],{"categories":4419},[170],{"categories":4421},[170],{"categories":4423},[97],{"categories":4425},[97],{"categories":4427},[],{"categories":4429},[],{"categories":4431},[195],{"categories":4433},[97],{"categories":4435},[170],{"categories":4437},[97],{"categories":4439},[244],{"categories":4441},[487],{"categories":4443},[431],{"categories":4445},[408],{"categories":4447},[97],{"categories":4449},[97],{"categories":4451},[97],{"categories":4453},[195],{"categories":4455},[97],{"categories":4457},[97],{"categories":4459},[97],{"categories":4461},[97],{"categories":4463},[97],{"categories":4465},[97],{"categories":4467},[97],{"categories":4469},[156],{"categories":4471},[146],{"categories":4473},[156],{"categories":4475},[97,151],{"categories":4477},[],{"categories":4479},[244],{"categories":4481},[],{"categories":4483},[159],{"categories":4485},[97],{"categories":4487},[192],{"categories":4489},[146],{"categories":4491},[97],{"categories":4493},[146],{"categories":4495},[156],{"categories":4497},[195],{"categories":4499},[156],{"categories":4501},[159],{"categories":4503},[156],{"categories":4505},[97],{"categories":4507},[97],{"categories":4509},[151],{"categories":4511},[156],{"categories":4513},[170],{"categories":4515},[273],{"categories":4517},[97],{"categories":4519},[97],{"categories":4521},[],{"categories":4523},[192],{"categories":4525},[97],{"categories":4527},[97],{"categories":4529},[97],{"categories":4531},[97],{"categories":4533},[97],{"categories":4535},[97],{"categories":4537},[170],{"categories":4539},[192],{"categories":4541},[170],{"categories":4543},[170],{"categories":4545},[97],{"categories":4547},[97],{"categories":4549},[97],{"categories":4551},[97],{"categories":4553},[431],{"categories":4555},[97],{"categories":4557},[156],{"categories":4559},[192],{"categories":4561},[97],{"categories":4563},[97],{"categories":4565},[97],{"categories":4567},[156],{"categories":4569},[97],{"categories":4571},[97],{"categories":4573},[97],{"categories":4575},[3166],{"categories":4577},[4578],"Clinical AI",{"categories":4580},[244],{"categories":4582},[97],{"categories":4584},[97],{"categories":4586},[97],{"categories":4588},[97],{"categories":4590},[314],{"categories":4592},[2525],{"categories":4594},[97],{"categories":4596},[159],{"categories":4598},[244],{"categories":4600},[97],{"categories":4602},[156],{"categories":4604},[97],{"categories":4606},[97],{"categories":4608},[192],{"categories":4610},[97],{"categories":4612},[156],{"categories":4614},[170],{"categories":4616},[273],{"categories":4618},[97],{"categories":4620},[97],{"categories":4622},[151],{"categories":4624},[97],{"categories":4626},[97],{"categories":4628},[546],{"categories":4630},[97],{"categories":4632},[],{"categories":4634},[156],{"categories":4636},[97],{"categories":4638},[170],{"categories":4640},[146],{"categories":4642},[97],{"categories":4644},[],{"categories":4646},[],{"categories":4648},[97],{"categories":4650},[],{"categories":4652},[151],{"categories":4654},[97],{"categories":4656},[97],{"categories":4658},[156],{"categories":4660},[97],{"categories":4662},[192],{"categories":4664},[192],{"categories":4666},[192],{"categories":4668},[192],{"categories":4670},[],{"categories":4672},[146],{"categories":4674},[156],{"categories":4676},[192],{"categories":4678},[97],{"categories":4680},[617],{"categories":4682},[159],{"categories":4684},[156],{"categories":4686},[97],{"categories":4688},[146],{"categories":4690},[97],{"categories":4692},[156],{"categories":4694},[97],{"categories":4696},[97],{"categories":4698},[97],{"categories":4700},[97,156],{"categories":4702},[156],{"categories":4704},[314],{"categories":4706},[192],{"categories":4708},[156],{"categories":4710},[192],{"categories":4712},[156],{"categories":4714},[97],{"categories":4716},[],{"categories":4718},[192],{"categories":4720},[273],{"categories":4722},[146],{"categories":4724},[97],{"categories":4726},[97],{"categories":4728},[],{"categories":4730},[170],{"categories":4732},[],{"categories":4734},[146],{"categories":4736},[156],{"categories":4738},[192],{"categories":4740},[97],{"categories":4742},[192],{"categories":4744},[146],{"categories":4746},[192],{"categories":4748},[192],{"categories":4750},[],{"categories":4752},[151],{"categories":4754},[156],{"categories":4756},[192],{"categories":4758},[192],{"categories":4760},[192],{"categories":4762},[192],{"categories":4764},[192],{"categories":4766},[192],{"categories":4768},[192],{"categories":4770},[192],{"categories":4772},[192],{"categories":4774},[192],{"categories":4776},[195],{"categories":4778},[146],{"categories":4780},[97],{"categories":4782},[97],{"categories":4784},[156],{"categories":4786},[156],{"categories":4788},[],{"categories":4790},[97],{"categories":4792},[97,146],{"categories":4794},[],{"categories":4796},[156],{"categories":4798},[97],{"categories":4800},[192],{"categories":4802},[156],{"categories":4804},[942],{"categories":4806},[97],{"categories":4808},[97],{"categories":4810},[97],{"categories":4812},[97],{"categories":4814},[97],{"categories":4816},[408],{"categories":4818},[97],{"categories":4820},[97],{"categories":4822},[156],{"categories":4824},[97],{"categories":4826},[97],{"categories":4828},[151],{"categories":4830},[159],{"categories":4832},[156],{"categories":4834},[156],{"categories":4836},[],{"categories":4838},[156],{"categories":4840},[244],{"categories":4842},[192],{"categories":4844},[97],{"categories":4846},[],{"categories":4848},[159],{"categories":4850},[],{"categories":4852},[170],{"categories":4854},[97],{"categories":4856},[156],{"categories":4858},[244],{"categories":4860},[97],{"categories":4862},[],{"categories":4864},[97],{"categories":4866},[97],{"categories":4868},[],{"categories":4870},[273],{"categories":4872},[97],{"categories":4874},[156],{"categories":4876},[],{"categories":4878},[],{"categories":4880},[192],{"categories":4882},[146],{"categories":4884},[97],{"categories":4886},[97],{"categories":4888},[151],{"categories":4890},[97],{"categories":4892},[97],{"categories":4894},[156],{"categories":4896},[97],{"categories":4898},[151],{"categories":4900},[151],{"categories":4902},[244],{"categories":4904},[],{"categories":4906},[97],{"categories":4908},[192],{"categories":4910},[],{"categories":4912},[97],{"categories":4914},[97],{"categories":4916},[244],{"categories":4918},[97],{"categories":4920},[97],{"categories":4922},[273],{"categories":4924},[97],{"categories":4926},[314],{"categories":4928},[],{"categories":4930},[156],{"categories":4932},[97],{"categories":4934},[273],{"categories":4936},[170],{"categories":4938},[],{"categories":4940},[97],{"categories":4942},[],{"categories":4944},[156],{"categories":4946},[244],{"categories":4948},[170],{"categories":4950},[],{"categories":4952},[3166],{"categories":4954},[151],{"categories":4956},[146],{"categories":4958},[97],{"categories":4960},[195],{"categories":4962},[156],{"categories":4964},[244],{"categories":4966},[97],{"categories":4968},[170],{"categories":4970},[],{"categories":4972},[],{"categories":4974},[97],{"categories":4976},[146],{"categories":4978},[97],{"categories":4980},[273],{"categories":4982},[],{"categories":4984},[156],{"categories":4986},[156],{"categories":4988},[97],{"categories":4990},[156],{"categories":4992},[97],{"categories":4994},[192],{"categories":4996},[170],{"categories":4998},[97],{"categories":5000},[156],{"categories":5002},[159],{"categories":5004},[97],{"categories":5006},[97],{"categories":5008},[97],{"categories":5010},[156],{"categories":5012},[97],{"categories":5014},[159],{"categories":5016},[273],{"categories":5018},[192],{"categories":5020},[],{"categories":5022},[273],{"categories":5024},[97],{"categories":5026},[],{"categories":5028},[170],{"categories":5030},[156],{"categories":5032},[],{"categories":5034},[97],{"categories":5036},[97],{"categories":5038},[97],{"categories":5040},[97],{"categories":5042},[97],{"categories":5044},[156],{"categories":5046},[151],{"categories":5048},[146],{"categories":5050},[156],{"categories":5052},[97],{"categories":5054},[244],{"categories":5056},[170],{"categories":5058},[170],{"categories":5060},[97],{"categories":5062},[195],{"categories":5064},[156],{"categories":5066},[97],{"categories":5068},[97],{"categories":5070},[156],{"categories":5072},[97],{"categories":5074},[97],{"categories":5076},[156],{"categories":5078},[151],{"categories":5080},[97],{"categories":5082},[244],{"categories":5084},[170],{"categories":5086},[156],{"categories":5088},[97],{"categories":5090},[159],{"categories":5092},[97],{"categories":5094},[156],{"categories":5096},[97],{"categories":5098},[97],{"categories":5100},[192],{"categories":5102},[97],{"categories":5104},[],{"categories":5106},[146],{"categories":5108},[97],{"categories":5110},[97],{"categories":5112},[97],{"categories":5114},[170],{"categories":5116},[170],{"categories":5118},[97],{"categories":5120},[170],{"categories":5122},[97],{"categories":5124},[156],{"categories":5126},[97],{"categories":5128},[97],{"categories":5130},[97],{"categories":5132},[97],{"categories":5134},[97],{"categories":5136},[],{"categories":5138},[97],{"categories":5140},[244],{"categories":5142},[156],{"categories":5144},[151],{"categories":5146},[192],{"categories":5148},[97],{"categories":5150},[156],{"categories":5152},[97],{"categories":5154},[156],{"categories":5156},[97],{"categories":5158},[97],{"categories":5160},[244],{"categories":5162},[156],{"categories":5164},[97],{"categories":5166},[273],{"categories":5168},[97],{"categories":5170},[195],{"categories":5172},[97],{"categories":5174},[97],{"categories":5176},[192],{"categories":5178},[97],{"categories":5180},[97],{"categories":5182},[97],{"categories":5184},[97],{"categories":5186},[156],{"categories":5188},[314],{"categories":5190},[97],{"categories":5192},[170],{"categories":5194},[156],{"categories":5196},[195],{"categories":5198},[],{"categories":5200},[156],{"categories":5202},[170],{"categories":5204},[97],{"categories":5206},[97],{"categories":5208},[2363],{"categories":5210},[244],{"categories":5212},[343],{"categories":5214},[97],{"categories":5216},[97],{"categories":5218},[97],{"categories":5220},[97],{"categories":5222},[146],{"categories":5224},[97],{"categories":5226},[97],{"categories":5228},[170],{"categories":5230},[151],{"categories":5232},[97],{"categories":5234},[170],{"categories":5236},[97],{"categories":5238},[],{"categories":5240},[156],{"categories":5242},[156],{"categories":5244},[97],{"categories":5246},[97],{"categories":5248},[97],{"categories":5250},[195],{"categories":5252},[],{"categories":5254},[192],{"categories":5256},[],{"categories":5258},[192],{"categories":5260},[97],{"categories":5262},[97],{"categories":5264},[156],{"categories":5266},[97],{"categories":5268},[156],{"categories":5270},[156],{"categories":5272},[],{"categories":5274},[97],{"categories":5276},[192],{"categories":5278},[97],{"categories":5280},[],{"categories":5282},[97],{"categories":5284},[97],{"categories":5286},[],{"categories":5288},[97],{"categories":5290},[97],{"categories":5292},[244],{"categories":5294},[170],{"categories":5296},[156],{"categories":5298},[97],{"categories":5300},[97],{"categories":5302},[97],{"categories":5304},[97],{"categories":5306},[273],{"categories":5308},[97],{"categories":5310},[97],{"categories":5312},[97],{"categories":5314},[146],{"categories":5316},[97],{"categories":5318},[97],{"categories":5320},[],{"categories":5322},[97],{"categories":5324},[97],{"categories":5326},[97],{"categories":5328},[],{"categories":5330},[146],{"categories":5332},[97],{"categories":5334},[97],{"categories":5336},[192],{"categories":5338},[170],{"categories":5340},[159],{"categories":5342},[156],{"categories":5344},[487],{"categories":5346},[97],{"categories":5348},[97],{"categories":5350},[97],{"categories":5352},[170],{"categories":5354},[192],{"categories":5356},[244],{"categories":5358},[97],{"categories":5360},[97],{"categories":5362},[97],{"categories":5364},[97],{"categories":5366},[192],{"categories":5368},[97],{"categories":5370},[244],{"categories":5372},[97],{"categories":5374},[97],{"categories":5376},[192],{"categories":5378},[244],{"categories":5380},[97],{"categories":5382},[192],{"categories":5384},[97],{"categories":5386},[156],{"categories":5388},[156],{"categories":5390},[156],{"categories":5392},[170],{"categories":5394},[192],{"categories":5396},[156],{"categories":5398},[156],{"categories":5400},[97],{"categories":5402},[170],{"categories":5404},[244],{"categories":5406},[97],{"categories":5408},[97],{"categories":5410},[156],{"categories":5412},[97],{"categories":5414},[],{"categories":5416},[156],{"categories":5418},[],{"categories":5420},[97],{"categories":5422},[97],{"categories":5424},[],{"categories":5426},[],{"categories":5428},[156],{"categories":5430},[151],{"categories":5432},[156],{"categories":5434},[5435],"Liability & Ethics",{"categories":5437},[97],{"categories":5439},[97],{"categories":5441},[97],{"categories":5443},[156],{"categories":5445},[146],{"categories":5447},[156],{"categories":5449},[151],{"categories":5451},[273],{"categories":5453},[156],{"categories":5455},[97],{"categories":5457},[97],{"categories":5459},[],{"categories":5461},[592],{"categories":5463},[156],{"categories":5465},[],{"categories":5467},[97],{"categories":5469},[146],{"categories":5471},[156],{"categories":5473},[],{"categories":5475},[156],{"categories":5477},[97],{"categories":5479},[97],{"categories":5481},[170],{"categories":5483},[97],{"categories":5485},[192],{"categories":5487},[97],{"categories":5489},[97],{"categories":5491},[159],{"categories":5493},[156],{"categories":5495},[97],{"categories":5497},[97],{"categories":5499},[97],{"categories":5501},[192],{"categories":5503},[156],{"categories":5505},[170],{"categories":5507},[244],{"categories":5509},[146],{"categories":5511},[97],{"categories":5513},[97],{"categories":5515},[97],{"categories":5517},[],{"categories":5519},[156],{"categories":5521},[156],{"categories":5523},[156],{"categories":5525},[487],{"categories":5527},[244],{"categories":5529},[156],{"categories":5531},[314],{"categories":5533},[170],{"categories":5535},[192],{"categories":5537},[97],{"categories":5539},[244],{"categories":5541},[97],{"categories":5543},[146],{"categories":5545},[],{"categories":5547},[156],{"categories":5549},[97],{"categories":5551},[97],{"categories":5553},[97],{"categories":5555},[97],{"categories":5557},[156],{"categories":5559},[97],{"categories":5561},[97],{"categories":5563},[244],{"categories":5565},[],{"categories":5567},[156],{"categories":5569},[159],{"categories":5571},[192],{"categories":5573},[156],{"categories":5575},[151],{"categories":5577},[],{"categories":5579},[97],{"categories":5581},[97],{"categories":5583},[159],{"categories":5585},[97],{"categories":5587},[156],{"categories":5589},[192],{"categories":5591},[146],{"categories":5593},[314],{"categories":5595},[97],{"categories":5597},[97],{"categories":5599},[97],{"categories":5601},[192],{"categories":5603},[151],{"categories":5605},[97],{"categories":5607},[244],{"categories":5609},[192],{"categories":5611},[314],{"categories":5613},[97],{"categories":5615},[156],{"categories":5617},[],{"categories":5619},[546],{"categories":5621},[],{"categories":5623},[97],{"categories":5625},[314],{"categories":5627},[97],{"categories":5629},[195],{"categories":5631},[97],{"categories":5633},[156],{"categories":5635},[156],{"categories":5637},[5638],"Design News & Tools",{"categories":5640},[97],{"categories":5642},[97],{"categories":5644},[192],{"categories":5646},[97],{"categories":5648},[97],{"categories":5650},[146],{"categories":5652},[156],{"categories":5654},[97],{"categories":5656},[244],{"categories":5658},[156],{"categories":5660},[156],{"categories":5662},[244],{"categories":5664},[97],{"categories":5666},[97],{"categories":5668},[487],{"categories":5670},[156],{"categories":5672},[97],{"categories":5674},[97],{"categories":5676},[487],{"categories":5678},[97],{"categories":5680},[273],{"categories":5682},[97],{"categories":5684},[156],{"categories":5686},[],{"categories":5688},[97],{"categories":5690},[97],{"categories":5692},[97],{"categories":5694},[192],{"categories":5696},[97],{"categories":5698},[146],{"categories":5700},[],{"categories":5702},[97],{"categories":5704},[97],{"categories":5706},[97],{"categories":5708},[170],{"categories":5710},[617],{"categories":5712},[170],{"categories":5714},[244],{"categories":5716},[97],{"categories":5718},[97,156],{"categories":5720},[273,151],{"categories":5722},[170],{"categories":5724},[97],{"categories":5726},[97],{"categories":5728},[97],{"categories":5730},[97],{"categories":5732},[],{"categories":5734},[156],{"categories":5736},[97],{"categories":5738},[],{"categories":5740},[97],{"categories":5742},[170],{"categories":5744},[97],{"categories":5746},[170],{"categories":5748},[],{"categories":5750},[156],{"categories":5752},[97],{"categories":5754},[151],{"categories":5756},[97],{"categories":5758},[192],{"categories":5760},[97],{"categories":5762},[],{"categories":5764},[156],{"categories":5766},[97],{"categories":5768},[],{"categories":5770},[244],{"categories":5772},[97],{"categories":5774},[97],{"categories":5776},[156],{"categories":5778},[97],{"categories":5780},[97],{"categories":5782},[146],{"categories":5784},[156],{"categories":5786},[97],{"categories":5788},[],{"categories":5790},[97],{"categories":5792},[314],{"categories":5794},[273],{"categories":5796},[151],{"categories":5798},[151],{"categories":5800},[97],{"categories":5802},[146],{"categories":5804},[146],{"categories":5806},[97],{"categories":5808},[156],{"categories":5810},[97],{"categories":5812},[97],{"categories":5814},[97],{"categories":5816},[97],{"categories":5818},[170],{"categories":5820},[97],{"categories":5822},[146],{"categories":5824},[97],{"categories":5826},[97],{"categories":5828},[156],{"categories":5830},[97],{"categories":5832},[273],{"categories":5834},[97],{"categories":5836},[192],{"categories":5838},[97],{"categories":5840},[97],{"categories":5842},[156],{"categories":5844},[159],{"categories":5846},[97],{"categories":5848},[97],{"categories":5850},[156],{"categories":5852},[],{"categories":5854},[170],{"categories":5856},[],{"categories":5858},[170],{"categories":5860},[156],{"categories":5862},[146],{"categories":5864},[97],{"categories":5866},[],{"categories":5868},[195],{"categories":5870},[314],{"categories":5872},[97],{"categories":5874},[170],{"categories":5876},[97],{"categories":5878},[],{"categories":5880},[192],{"categories":5882},[156],{"categories":5884},[170],{"categories":5886},[244],{"categories":5888},[151],{"categories":5890},[97],{"categories":5892},[97],{"categories":5894},[156],{"categories":5896},[170],{"categories":5898},[156],{"categories":5900},[192],{"categories":5902},[97],{"categories":5904},[159],{"categories":5906},[146],{"categories":5908},[159],{"categories":5910},[192],{"categories":5912},[97],{"categories":5914},[170],{"categories":5916},[97],{"categories":5918},[244],{"categories":5920},[151],{"categories":5922},[97],{"categories":5924},[97],{"categories":5926},[97],{"categories":5928},[97],{"categories":5930},[97],{"categories":5932},[97],{"categories":5934},[156],{"categories":5936},[97],{"categories":5938},[156],{"categories":5940},[97],{"categories":5942},[97],{"categories":5944},[146],{"categories":5946},[97],{"categories":5948},[156],{"categories":5950},[156],{"categories":5952},[244],{"categories":5954},[156],{"categories":5956},[156],{"categories":5958},[97],{"categories":5960},[146],{"categories":5962},[156],{"categories":5964},[244],{"categories":5966},[],{"categories":5968},[97],{"categories":5970},[195],{"categories":5972},[487],{"categories":5974},[97],{"categories":5976},[156],{"categories":5978},[97],{"categories":5980},[97],{"categories":5982},[170],{"categories":5984},[97],{"categories":5986},[],{"categories":5988},[97],{"categories":5990},[156],{"categories":5992},[97],{"categories":5994},[273],{"categories":5996},[97],{"categories":5998},[170],{"categories":6000},[97],{"categories":6002},[192],{"categories":6004},[156],{"categories":6006},[97],{"categories":6008},[273],{"categories":6010},[156],{"categories":6012},[151],{"categories":6014},[151],{"categories":6016},[97],{"categories":6018},[97],{"categories":6020},[97],{"categories":6022},[97],{"categories":6024},[97],{"categories":6026},[97],{"categories":6028},[146],{"categories":6030},[],{"categories":6032},[97],{"categories":6034},[97],{"categories":6036},[156],{"categories":6038},[156],{"categories":6040},[97],{"categories":6042},[97],{"categories":6044},[97],{"categories":6046},[97],{"categories":6048},[97],{"categories":6050},[170],{"categories":6052},[],{"categories":6054},[146],{"categories":6056},[97],{"categories":6058},[97],{"categories":6060},[156],{"categories":6062},[156],{"categories":6064},[],{"categories":6066},[170],{"categories":6068},[170],{"categories":6070},[97],{"categories":6072},[273],{"categories":6074},[151],{"categories":6076},[244],{"categories":6078},[],{"categories":6080},[97],{"categories":6082},[156],{"categories":6084},[146],{"categories":6086},[97],{"categories":6088},[97],{"categories":6090},[170],{"categories":6092},[146],{"categories":6094},[97],{"categories":6096},[97],{"categories":6098},[192],{"categories":6100},[195],{"categories":6102},[97],{"categories":6104},[192],{"categories":6106},[156],{"categories":6108},[97],{"categories":6110},[],{"categories":6112},[192],{"categories":6114},[156],{"categories":6116},[244],{"categories":6118},[195],{"categories":6120},[97],{"categories":6122},[97],{"categories":6124},[],{"categories":6126},[156],{"categories":6128},[156],{"categories":6130},[156],{"categories":6132},[3166],{"categories":6134},[192],{"categories":6136},[97],{"categories":6138},[170],{"categories":6140},[97],{"categories":6142},[97],{"categories":6144},[97],{"categories":6146},[97],{"categories":6148},[97],{"categories":6150},[151],{"categories":6152},[97],{"categories":6154},[146],{"categories":6156},[1829],{"categories":6158},[314],{"categories":6160},[146],{"categories":6162},[],{"categories":6164},[97],{"categories":6166},[],{"categories":6168},[192],{"categories":6170},[156],{"categories":6172},[244],{"categories":6174},[97],{"categories":6176},[97],{"categories":6178},[97],{"categories":6180},[192],{"categories":6182},[],{"categories":6184},[156],{"categories":6186},[97],{"categories":6188},[156],{"categories":6190},[156],{"categories":6192},[],{"categories":6194},[97],{"categories":6196},[],{"categories":6198},[192],{"categories":6200},[146],{"categories":6202},[244],{"categories":6204},[97],{"categories":6206},[156],{"categories":6208},[192],{"categories":6210},[97],{"categories":6212},[192],{"categories":6214},[],{"categories":6216},[192],{"categories":6218},[97],{"categories":6220},[146],{"categories":6222},[487],{"categories":6224},[156],{"categories":6226},[97],{"categories":6228},[],{"categories":6230},[170],{"categories":6232},[156],{"categories":6234},[159],{"categories":6236},[156],{"categories":6238},[146],{"categories":6240},[97],{"categories":6242},[97],{"categories":6244},[],{"categories":6246},[],{"categories":6248},[],{"categories":6250},[244],{"categories":6252},[97],{"categories":6254},[156],{"categories":6256},[97],{"categories":6258},[97],{"categories":6260},[],{"categories":6262},[],{"categories":6264},[],{"categories":6266},[97],{"categories":6268},[156],{"categories":6270},[244],{"categories":6272},[97],{"categories":6274},[],{"categories":6276},[156],{"categories":6278},[97],{"categories":6280},[97],{"categories":6282},[146],{"categories":6284},[],{"categories":6286},[],{"categories":6288},[97],{"categories":6290},[97],{"categories":6292},[156],{"categories":6294},[244],{"categories":6296},[97],{"categories":6298},[192],{"categories":6300},[],{"categories":6302},[97],{"categories":6304},[97],{"categories":6306},[273],{"categories":6308},[192],{"categories":6310},[273],{"categories":6312},[195],{"categories":6314},[97],{"categories":6316},[97],{"categories":6318},[],{"categories":6320},[],{"categories":6322},[156],{"categories":6324},[],{"categories":6326},[97],{"categories":6328},[487],{"categories":6330},[97],{"categories":6332},[97],{"categories":6334},[97],{"categories":6336},[97],{"categories":6338},[],{"categories":6340},[156],{"categories":6342},[97],{"categories":6344},[97],{"categories":6346},[],{"categories":6348},[156],{"categories":6350},[97],{"categories":6352},[192],{"categories":6354},[97],{"categories":6356},[273],{"categories":6358},[151],{"categories":6360},[159],{"categories":6362},[97],{"categories":6364},[97],{"categories":6366},[156],{"categories":6368},[195],{"categories":6370},[156],{"categories":6372},[156],{"categories":6374},[],{"categories":6376},[97],{"categories":6378},[156],{"categories":6380},[],{"categories":6382},[97],{"categories":6384},[],{"categories":6386},[192],{"categories":6388},[151],{"categories":6390},[],{"categories":6392},[97],{"categories":6394},[97],{"categories":6396},[97],{"categories":6398},[],{"categories":6400},[156],{"categories":6402},[244],{"categories":6404},[146],{"categories":6406},[97],{"categories":6408},[],{"categories":6410},[151],{"categories":6412},[273],{"categories":6414},[97],{"categories":6416},[170],{"categories":6418},[146],{"categories":6420},[195],{"categories":6422},[151],{"categories":6424},[170],{"categories":6426},[156],{"categories":6428},[170],{"categories":6430},[],{"categories":6432},[97],{"categories":6434},[159],{"categories":6436},[97],{"categories":6438},[],{"categories":6440},[156],{"categories":6442},[146],{"categories":6444},[244],{"categories":6446},[97],{"categories":6448},[146],{"categories":6450},[156],{"categories":6452},[314],{"categories":6454},[97],{"categories":6456},[97],{"categories":6458},[97],{"categories":6460},[97],{"categories":6462},[97],{"categories":6464},[146],{"categories":6466},[97],{"categories":6468},[170],{"categories":6470},[195],{"categories":6472},[156],{"categories":6474},[],{"categories":6476},[97],{"categories":6478},[97],{"categories":6480},[97],{"categories":6482},[170],{"categories":6484},[156],{"categories":6486},[192],{"categories":6488},[170],{"categories":6490},[97],{"categories":6492},[159],{"categories":6494},[],{"categories":6496},[244],{"categories":6498},[170],{"categories":6500},[192],{"categories":6502},[97],{"categories":6504},[146],{"categories":6506},[156],{"categories":6508},[97],{"categories":6510},[97],{"categories":6512},[156],{"categories":6514},[159],{"categories":6516},[97],{"categories":6518},[156],{"categories":6520},[97],{"categories":6522},[151],{"categories":6524},[156],{"categories":6526},[156,314],{"categories":6528},[97],{"categories":6530},[97],{"categories":6532},[156],{"categories":6534},[170],{"categories":6536},[97],{"categories":6538},[97],{"categories":6540},[195],{"categories":6542},[156],{"categories":6544},[273],{"categories":6546},[156],{"categories":6548},[151],{"categories":6550},[],{"categories":6552},[156],{"categories":6554},[97],{"categories":6556},[151],{"categories":6558},[],{"categories":6560},[],{"categories":6562},[170],{"categories":6564},[97],{"categories":6566},[97],{"categories":6568},[156],{"categories":6570},[195],{"categories":6572},[273],{"categories":6574},[97],{"categories":6576},[97],{"categories":6578},[97],{"categories":6580},[156],{"categories":6582},[],{"categories":6584},[156],{"categories":6586},[192],{"categories":6588},[97],{"categories":6590},[156],{"categories":6592},[156],{"categories":6594},[97],{"categories":6596},[],{"categories":6598},[192],{"categories":6600},[170],{"categories":6602},[3166],{"categories":6604},[146],{"categories":6606},[170],{"categories":6608},[97],{"categories":6610},[156],{"categories":6612},[97],{"categories":6614},[97],{"categories":6616},[273],{"categories":6618},[170],{"categories":6620},[195],{"categories":6622},[],{"categories":6624},[192],{"categories":6626},[97],{"categories":6628},[97],{"categories":6630},[],{"categories":6632},[156],{"categories":6634},[97],{"categories":6636},[97],{"categories":6638},[97],{"categories":6640},[97],{"categories":6642},[156],{"categories":6644},[97],{"categories":6646},[97],{"categories":6648},[97],{"categories":6650},[159],{"categories":6652},[97],{"categories":6654},[156],{"categories":6656},[97],{"categories":6658},[97],{"categories":6660},[97],{"categories":6662},[97],{"categories":6664},[97],{"categories":6666},[97],{"categories":6668},[97],{"categories":6670},[151],{"categories":6672},[],{"categories":6674},[159],{"categories":6676},[192],{"categories":6678},[156],{"categories":6680},[97],{"categories":6682},[170],{"categories":6684},[],{"categories":6686},[170],{"categories":6688},[170],{"categories":6690},[156],{"categories":6692},[170],{"categories":6694},[97],{"categories":6696},[97],{"categories":6698},[97],{"categories":6700},[156],{"categories":6702},[170],{"categories":6704},[97],{"categories":6706},[97],{"categories":6708},[97],{"categories":6710},[156],{"categories":6712},[192],{"categories":6714},[97],{"categories":6716},[97],{"categories":6718},[97],{"categories":6720},[151],{"categories":6722},[97],{"categories":6724},[156],{"categories":6726},[244],{"categories":6728},[],{"categories":6730},[97],{"categories":6732},[195],{"categories":6734},[156],{"categories":6736},[97],{"categories":6738},[97],{"categories":6740},[],{"categories":6742},[97],{"categories":6744},[97],{"categories":6746},[192],{"categories":6748},[97],{"categories":6750},[97],{"categories":6752},[156],{"categories":6754},[273],{"categories":6756},[],{"categories":6758},[],{"categories":6760},[170],{"categories":6762},[97],{"categories":6764},[97],{"categories":6766},[192],{"categories":6768},[97],{"categories":6770},[170],{"categories":6772},[192],{"categories":6774},[97],{"categories":6776},[97],{"categories":6778},[273],{"categories":6780},[195],{"categories":6782},[97],{"categories":6784},[97],{"categories":6786},[146],{"categories":6788},[156],{"categories":6790},[97],{"categories":6792},[97],{"categories":6794},[156],{"categories":6796},[151],{"categories":6798},[156],{"categories":6800},[170],{"categories":6802},[97],{"categories":6804},[151],{"categories":6806},[],{"categories":6808},[97],{"categories":6810},[195],{"categories":6812},[97],{"categories":6814},[97],{"categories":6816},[],{"categories":6818},[192],{"categories":6820},[97],{"categories":6822},[156],{"categories":6824},[195],{"categories":6826},[97],{"categories":6828},[170],{"categories":6830},[170],{"categories":6832},[170],{"categories":6834},[97],{"categories":6836},[156],{"categories":6838},[156],{"categories":6840},[97],{"categories":6842},[156],{"categories":6844},[97],{"categories":6846},[97],{"categories":6848},[244],{"categories":6850},[195],{"categories":6852},[195],{"categories":6854},[],{"categories":6856},[192],{"categories":6858},[97],{"categories":6860},[97],{"categories":6862},[170],{"categories":6864},[],{"categories":6866},[192],{"categories":6868},[192],{"categories":6870},[192],{"categories":6872},[],{"categories":6874},[156],{"categories":6876},[97],{"categories":6878},[],{"categories":6880},[146],{"categories":6882},[151],{"categories":6884},[],{"categories":6886},[97],{"categories":6888},[97],{"categories":6890},[],{"categories":6892},[170],{"categories":6894},[],{"categories":6896},[],{"categories":6898},[],{"categories":6900},[],{"categories":6902},[97],{"categories":6904},[192],{"categories":6906},[],{"categories":6908},[],{"categories":6910},[97],{"categories":6912},[97],{"categories":6914},[97],{"categories":6916},[195],{"categories":6918},[97],{"categories":6920},[195],{"categories":6922},[],{"categories":6924},[195],{"categories":6926},[195],{"categories":6928},[314],{"categories":6930},[156],{"categories":6932},[170],{"categories":6934},[],{"categories":6936},[],{"categories":6938},[195],{"categories":6940},[170],{"categories":6942},[170],{"categories":6944},[170],{"categories":6946},[],{"categories":6948},[146],{"categories":6950},[170],{"categories":6952},[170],{"categories":6954},[146],{"categories":6956},[170],{"categories":6958},[151],{"categories":6960},[170],{"categories":6962},[170],{"categories":6964},[170],{"categories":6966},[195],{"categories":6968},[192],{"categories":6970},[192],{"categories":6972},[97],{"categories":6974},[170],{"categories":6976},[195],{"categories":6978},[314],{"categories":6980},[195],{"categories":6982},[195],{"categories":6984},[195],{"categories":6986},[],{"categories":6988},[151],{"categories":6990},[],{"categories":6992},[314],{"categories":6994},[170],{"categories":6996},[170],{"categories":6998},[170],{"categories":7000},[156],{"categories":7002},[192,151],{"categories":7004},[195],{"categories":7006},[],{"categories":7008},[],{"categories":7010},[195],{"categories":7012},[],{"categories":7014},[195],{"categories":7016},[192],{"categories":7018},[156],{"categories":7020},[],{"categories":7022},[170],{"categories":7024},[97],{"categories":7026},[244],{"categories":7028},[],{"categories":7030},[97],{"categories":7032},[],{"categories":7034},[192],{"categories":7036},[146],{"categories":7038},[195],{"categories":7040},[],{"categories":7042},[170],{"categories":7044},[192],[7046,7183,7272,7419],{"id":7047,"title":7048,"ai":7049,"body":7054,"categories":7146,"created_at":98,"date_modified":98,"description":90,"extension":99,"faq":98,"featured":100,"kicker_label":98,"meta":7147,"navigation":120,"path":7168,"published_at":7169,"question":98,"scraped_at":7170,"seo":7171,"sitemap":7172,"source_id":7173,"source_name":127,"source_type":128,"source_url":7174,"stem":7175,"tags":7176,"thumbnail_url":7178,"tldr":7179,"tweet":7180,"unknown_tags":7181,"__hash__":7182},"summaries\u002Fsummaries\u002F85298428c0b7fdc7-practical-evaluation-strategies-for-ai-agents-summary.md","Practical Evaluation Strategies for AI Agents",{"provider":7,"model":8,"input_tokens":7050,"output_tokens":7051,"processing_time_ms":7052,"cost_usd":7053},8160,851,4446,0.0033165,{"type":14,"value":7055,"toc":7141},[7056,7060,7063,7066,7086,7090,7093,7114,7118,7121],[17,7057,7059],{"id":7058},"the-philosophy-of-evals-beyond-the-leaderboard","The Philosophy of Evals: Beyond the Leaderboard",[22,7061,7062],{},"Evaluation metrics are often misunderstood. They are neither objective truths nor useless vibes, but rather a tool for iterative engineering. The core problem with current benchmarks is that they are often outdated or optimized for \"benchmark maxing\" rather than real-world utility.",[22,7064,7065],{},"To effectively use evals, adopt three heuristics:",[38,7067,7068,7074,7080],{},[41,7069,7070,7073],{},[44,7071,7072],{},"Be Skeptical:"," Treat model-provided benchmark numbers as approximations. They rarely capture the nuance of real-world performance.",[41,7075,7076,7079],{},[44,7077,7078],{},"Avoid Bleeding Edge:"," Let new model releases \"set on fire\" for a few weeks. Only switch models once they have proven their stability and utility in production environments.",[41,7081,7082,7085],{},[44,7083,7084],{},"Prioritize Precision:"," Use benchmarks that measure actual capabilities (e.g., complex engineering tasks) rather than standardized tests that no longer reflect frontier performance.",[17,7087,7089],{"id":7088},"the-hill-climbing-framework","The Hill-Climbing Framework",[22,7091,7092],{},"Improving an agent is an engineering and philosophy problem. Because agentic workflows involve complex, multi-step processes (searching docs, installing environments, running code), you cannot rely on simple binary evals. Instead, implement a systematic hill-climbing process:",[7094,7095,7096,7102,7108],"ol",{},[41,7097,7098,7101],{},[44,7099,7100],{},"Environment Isolation:"," Run tasks in isolated, standardized environments (e.g., using Docker or virtual machines) to ensure reproducibility.",[41,7103,7104,7107],{},[44,7105,7106],{},"Parallelization:"," Use infrastructure like Modal or Harbor to run large test suites in parallel, making the slowest task the only limiting factor.",[41,7109,7110,7113],{},[44,7111,7112],{},"Portfolio Allocation of Failures:"," After a run, use a secondary agent to analyze the failure traces. This identifies which specific levers—such as container CPU\u002Fmemory settings, timeout adjustments, or model-specific prompt engineering—actually move the needle.",[17,7115,7117],{"id":7116},"categorizing-improvements","Categorizing Improvements",[22,7119,7120],{},"When analyzing your results, group your findings into three zones:",[38,7122,7123,7129,7135],{},[41,7124,7125,7128],{},[44,7126,7127],{},"Zone 1 (Obvious Flaws):"," Fix clear bugs, rate-limiting issues, or harness crashes.",[41,7130,7131,7134],{},[44,7132,7133],{},"Zone 2 (Nuance Improvements):"," This is the most critical area. It involves tailoring prompt engineering and harness logic to specific model families (e.g., Anthropic vs. Gemini). This explains why a model might perform exceptionally well in a general sense but fail in your specific harness.",[41,7136,7137,7140],{},[44,7138,7139],{},"Zone 3 (The Danger Zone):"," Avoid overfitting to the benchmark. Cheating to get a higher score for a tweet provides no value to the end user and degrades the product's actual intelligence.",{"title":90,"searchDepth":91,"depth":91,"links":7142},[7143,7144,7145],{"id":7058,"depth":91,"text":7059},{"id":7088,"depth":91,"text":7089},{"id":7116,"depth":91,"text":7117},[97],{"content_references":7148,"triage":7164},[7149,7152,7155,7158,7161],{"type":110,"title":7150,"url":7151,"context":114},"Terminal Bench","https:\u002F\u002Fgithub.com\u002Flott-institute\u002Fterminal-bench",{"type":110,"title":7153,"url":7154,"context":114},"Harbor","https:\u002F\u002Fgithub.com\u002Flott-institute\u002Fharbor",{"type":110,"title":7156,"url":7157,"context":114},"Modal","https:\u002F\u002Fmodal.com\u002F",{"type":110,"title":7159,"url":7160,"context":112},"Daytona","https:\u002F\u002Fwww.daytona.io\u002F",{"type":110,"title":7162,"url":7163,"context":112},"Cline","https:\u002F\u002Fgithub.com\u002Fcline\u002Fcline",{"relevance":7165,"novelty":116,"quality":116,"actionability":7165,"composite":7166,"reasoning":7167},5,4.55,"Category: AI & LLMs. The article provides a deep dive into practical evaluation strategies for AI agents, addressing the audience's need for actionable insights on improving AI performance. It outlines specific heuristics and a systematic framework for evaluating and enhancing agent capabilities, making it highly relevant and actionable for product builders.","\u002Fsummaries\u002F85298428c0b7fdc7-practical-evaluation-strategies-for-ai-agents-summary","2026-06-06 16:00:00","2026-06-06 16:08:37",{"title":7048,"description":90},{"loc":7168},"85298428c0b7fdc7","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=QuuIywMG4s8","summaries\u002F85298428c0b7fdc7-practical-evaluation-strategies-for-ai-agents-summary",[7177,132,134,135],"llm","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FQuuIywMG4s8\u002Fhqdefault.jpg","Benchmark numbers are not gospel, but they are essential for iterative improvement. Use them to hill-climb your agent's performance by identifying failure patterns rather than chasing leaderboard scores.","This talk argues that public benchmarks are unreliable for evaluating agentic workflows, proposing instead a \"failure-analysis\" framework: after running an agent, use a secondary agent to categorize failure traces into bugs, nuance-based performance gaps, or benchmark overfitting.",[132,134,135],"GTMZr0m82TN_vhKnpVYbrcfP7gwvKRy8_Z5wv7jHPAQ",{"id":7184,"title":7185,"ai":7186,"body":7191,"categories":7247,"created_at":98,"date_modified":98,"description":90,"extension":99,"faq":98,"featured":100,"kicker_label":98,"meta":7248,"navigation":120,"path":7256,"published_at":7257,"question":98,"scraped_at":7258,"seo":7259,"sitemap":7260,"source_id":7261,"source_name":7262,"source_type":128,"source_url":7263,"stem":7264,"tags":7265,"thumbnail_url":7267,"tldr":7268,"tweet":7269,"unknown_tags":7270,"__hash__":7271},"summaries\u002Fsummaries\u002F34d109b3ae5f0509-ai-agents-why-the-harness-matters-more-than-the-mo-summary.md","AI Agents: Why the Harness Matters More Than the Model",{"provider":7,"model":8,"input_tokens":7187,"output_tokens":7188,"processing_time_ms":7189,"cost_usd":7190},5246,692,3597,0.0023495,{"type":14,"value":7192,"toc":7242},[7193,7197,7204,7208,7211,7235,7239],[17,7194,7196],{"id":7195},"the-brain-vs-the-jar","The Brain vs. The Jar",[22,7198,7199,7200,7203],{},"AI models (like GPT-4 or Claude) are essentially 'brains in a jar.' While they are highly capable, they are inherently limited by their inability to interact with the outside world—they cannot natively run code, browse the web, or access files. The performance gap between AI products often isn't due to the underlying model, as benchmark scores between top labs have converged. Instead, the difference lies in the ",[44,7201,7202],{},"agentic harness",": the architectural wrapper that enables the model to act, remember, and iterate.",[17,7205,7207],{"id":7206},"the-three-pillars-of-the-agentic-harness","The Three Pillars of the Agentic Harness",[22,7209,7210],{},"To transform a static model into a functional agent, the harness must provide three critical capabilities:",[38,7212,7213,7219,7229],{},[41,7214,7215,7218],{},[44,7216,7217],{},"Tools:"," These allow the model to interface with the world. Beyond basic file I\u002FO, modern harnesses enable code execution in sandboxes, web browsing, and 'computer use' (controlling a cursor to operate software). The Model Context Protocol (MCP) is emerging as a standard to allow these tools to plug into various harnesses without custom rebuilds.",[41,7220,7221,7224,7225,7228],{},[44,7222,7223],{},"Memory:"," Because models have finite context windows that reset after a session, the harness must manage persistence. This includes loading instruction files (e.g., ",[26,7226,7227],{},"agents.md",") to define project conventions, compacting context windows by summarizing past interactions, and implementing retrieval systems (semantic search or code indexing) to feed the model only the relevant data rather than the entire codebase.",[41,7230,7231,7234],{},[44,7232,7233],{},"Agentic Loops:"," This is the operational cycle where the model plans a step, the harness executes it, and the model observes the result. Sophisticated harnesses add a verification layer to this loop, such as running automated tests, taking screenshots of UI states, or using a secondary 'reviewer' model to ensure the agent stays on track during long-running tasks.",[17,7236,7238],{"id":7237},"the-fluid-future-of-ai-architecture","The Fluid Future of AI Architecture",[22,7240,7241],{},"It is a mistake to evaluate AI capability based solely on the model. A brilliant model can fail in a poorly designed harness, while a mid-tier model can perform exceptionally well in a robust one. The distinction between the two is becoming increasingly fluid: complex behaviors like long-horizon planning are being trained directly into models, while consistency and project-specific knowledge are increasingly managed by harness-level conventions. Builders should focus on optimizing the 'jar' as much as the 'brain' to achieve production-grade results.",{"title":90,"searchDepth":91,"depth":91,"links":7243},[7244,7245,7246],{"id":7195,"depth":91,"text":7196},{"id":7206,"depth":91,"text":7207},{"id":7237,"depth":91,"text":7238},[97],{"content_references":7249,"triage":7253},[7250],{"type":110,"title":7251,"url":7252,"context":112},"Model Context Protocol (MCP)","https:\u002F\u002Fmodelcontextprotocol.io\u002F",{"relevance":7165,"novelty":116,"quality":116,"actionability":116,"composite":7254,"reasoning":7255},4.35,"Category: AI & LLMs. The article provides a deep dive into the concept of the 'agentic harness' which is crucial for building effective AI agents, addressing a key pain point for developers looking to integrate AI into their products. It outlines specific capabilities of the harness, such as tools, memory management, and operational loops, which are actionable insights for product builders.","\u002Fsummaries\u002F34d109b3ae5f0509-ai-agents-why-the-harness-matters-more-than-the-mo-summary","2026-08-25 11:00:31","2026-08-26 03:09:58",{"title":7185,"description":90},{"loc":7256},"34d109b3ae5f0509","IBM Technology","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=ZELPNFXJ4_o","summaries\u002F34d109b3ae5f0509-ai-agents-why-the-harness-matters-more-than-the-mo-summary",[7177,7266,132,135],"automation","https:\u002F\u002Fi.ytimg.com\u002Fvi\u002FZELPNFXJ4_o\u002Fhqdefault.jpg","AI system performance is driven by the 'agentic harness'—the tools, memory, and loops surrounding the model—rather than just the model itself. Distinguishing between the 'brain' (model) and the 'jar' (harness) is essential for building effective AI agents.","This video provides a conceptual breakdown of why AI performance often depends more on the \"agentic harness\"—the tools, memory, and loops wrapped around a model—than on the model itself. It is a high-level primer on how [AI models](https:\u002F\u002Fibm.biz\u002F~fAH1kt7jT) are integrated into functional systems.",[132,135],"PtVTsoTGi-qI9ZNh4s66ldbH7fWoQkYBn7wif1qsZ-o",{"id":7273,"title":7274,"ai":7275,"body":7280,"categories":7384,"created_at":98,"date_modified":98,"description":90,"extension":99,"faq":98,"featured":100,"kicker_label":98,"meta":7385,"navigation":120,"path":7405,"published_at":7406,"question":98,"scraped_at":7406,"seo":7407,"sitemap":7408,"source_id":7409,"source_name":7410,"source_type":7411,"source_url":7412,"stem":7413,"tags":7414,"thumbnail_url":98,"tldr":7416,"tweet":98,"unknown_tags":7417,"__hash__":7418},"summaries\u002Fsummaries\u002F932c42c2bd21ef48-human-judgment-in-the-age-of-ai-software-factories-summary.md","Human Judgment in the Age of AI Software Factories",{"provider":7,"model":8,"input_tokens":7276,"output_tokens":7277,"processing_time_ms":7278,"cost_usd":7279},8889,887,3730,0.00355275,{"type":14,"value":7281,"toc":7378},[7282,7286,7289,7293,7296,7299,7344,7348,7351,7371,7375],[17,7283,7285],{"id":7284},"the-shift-from-coding-to-curation","The Shift from Coding to Curation",[22,7287,7288],{},"As AI agents enable parallel development, human cognitive bandwidth becomes the primary bottleneck. Rather than attempting to review every line of generated code, engineers must shift their focus upstream to defining product intent and system architecture, and downstream to verifying outcomes. Human judgment hasn't disappeared; it has relocated to where it provides the most leverage: setting quality bars, defining constraints, and deciding which projects deserve to exist.",[17,7290,7292],{"id":7291},"building-a-repeatable-software-factory","Building a Repeatable Software Factory",[22,7294,7295],{},"A software factory is an event-driven loop (triggered by tools like GitHub issues or Slack) that handles triage, implementation, and testing in isolated environments. You should consider building one when your workflow requires consistency across multiple agents and projects.",[22,7297,7298],{},"Key components of a robust factory include:",[38,7300,7301,7315,7321],{},[41,7302,7303,7306,7307,7310,7311,7314],{},[44,7304,7305],{},"Triage Labels:"," Using labels (e.g., ",[26,7308,7309],{},"ready-to-implement",", ",[26,7312,7313],{},"needs-info",") as a queue and lock mechanism to prevent multiple agents from claiming the same task.",[41,7316,7317,7320],{},[44,7318,7319],{},"Verification Budgets:"," Similar to performance budgets, you must balance fast, deterministic checks (linting, type checking) early in the loop with heavier, high-value checks (mutation testing, security scans) later.",[41,7322,7323,7326,7327,7310,7330,7310,7333,7336,7337,7340,7341,7343],{},[44,7324,7325],{},"Run Taxonomy:"," Classify agent runs as ",[26,7328,7329],{},"success",[26,7331,7332],{},"flawed",[26,7334,7335],{},"blocked",", or ",[26,7338,7339],{},"manual",". Only ",[26,7342,7329],{}," should reach production, while the others re-enter the system for correction or human intervention.",[17,7345,7347],{"id":7346},"managing-risk-and-comprehension-debt","Managing Risk and Comprehension Debt",[22,7349,7350],{},"Parallel agent execution creates 'comprehension debt'—where agent output outpaces human understanding. To mitigate this:",[38,7352,7353,7359,7365],{},[41,7354,7355,7358],{},[44,7356,7357],{},"Avoid 'Green' Misconceptions:"," Agents may pass tests by modifying assertions rather than satisfying intent. Ensure your verification suite is robust and tests actual behavior.",[41,7360,7361,7364],{},[44,7362,7363],{},"Optimize for the Reviewer:"," A factory is only as good as the signal it provides. If a factory runs noisy, irrelevant checks, it increases overhead without improving safety.",[41,7366,7367,7370],{},[44,7368,7369],{},"Contextual Autonomy:"," Autonomy is not a static setting. As you build trust through verification, you can grant agents more autonomy for specific tasks, but you must remain the final arbiter for high-risk changes (e.g., authentication, billing, or migrations).",[17,7372,7374],{"id":7373},"when-to-build-vs-use","When to Build vs. Use",[22,7376,7377],{},"For many, a simple coding harness (e.g., Claude Code with well-defined specs) is sufficient. A full software factory is only necessary when you need to automate the handoff between triage, implementation, and production monitoring. If you decide to build, consider existing platforms like Factory, Warp, or HumanLayer rather than building from scratch.",{"title":90,"searchDepth":91,"depth":91,"links":7379},[7380,7381,7382,7383],{"id":7284,"depth":91,"text":7285},{"id":7291,"depth":91,"text":7292},{"id":7346,"depth":91,"text":7347},{"id":7373,"depth":91,"text":7374},[97],{"group":7386,"content_references":7387,"triage":7403},"addy-osmani",[7388,7390,7393,7396,7399],{"type":110,"title":7389,"context":112},"Claude Code",{"type":110,"title":7391,"url":7392,"context":114},"Factory","https:\u002F\u002Ffactory.com\u002F",{"type":110,"title":7394,"url":7395,"context":114},"Warp","https:\u002F\u002Fwww.warp.dev\u002F",{"type":110,"title":7397,"url":7398,"context":114},"HumanLayer","https:\u002F\u002Fwww.humanlayer.dev\u002F",{"type":110,"title":7400,"url":7401,"context":7402},"Vercel AI SDK","https:\u002F\u002Fvercel.com\u002Fblog\u002Fbuilding-a-software-factory-for-ai-sdk","cited",{"relevance":7165,"novelty":116,"quality":116,"actionability":116,"composite":7254,"reasoning":7404},"Category: Software Engineering. The article provides a detailed framework for building a 'software factory' that integrates AI agents into the development process, addressing the audience's need for practical applications in AI-powered product development. It outlines specific components like triage labels and verification budgets, making it actionable for developers and product builders.","\u002Fsummaries\u002F932c42c2bd21ef48-human-judgment-in-the-age-of-ai-software-factories-summary","2026-08-24 03:08:32",{"title":7274,"description":90},{"loc":7405},"932c42c2bd21ef48","Addy Osmani Blog","article","https:\u002F\u002Faddyosmani.com\u002Fblog\u002Fhuman-judgment-doesnt-leave-the-software\u002F","summaries\u002F932c42c2bd21ef48-human-judgment-in-the-age-of-ai-software-factories-summary",[7266,7415,132,135],"product-strategy","As AI agents scale development, human judgment shifts from writing code to defining intent, system design, and verification strategy. A 'software factory'—a repeatable, event-driven loop—is the framework for managing this shift, provided you balance verification budgets with human oversight.",[132,135],"Vo6frxSFOkz_YA1cBgEljyvwSFoayJFe86TBrUphKaE",{"id":7420,"title":7421,"ai":7422,"body":7427,"categories":7490,"created_at":98,"date_modified":98,"description":90,"extension":99,"faq":98,"featured":100,"kicker_label":98,"meta":7491,"navigation":120,"path":7504,"published_at":7505,"question":98,"scraped_at":7506,"seo":7507,"sitemap":7508,"source_id":7509,"source_name":127,"source_type":128,"source_url":7510,"stem":7511,"tags":7512,"thumbnail_url":98,"tldr":7515,"tweet":7516,"unknown_tags":7517,"__hash__":7518},"summaries\u002Fsummaries\u002F5bc49edfb5f3bc68-raising-the-floor-practical-ai-agent-evaluation-summary.md","Raising the Floor: Practical AI Agent Evaluation",{"provider":7,"model":8,"input_tokens":7423,"output_tokens":7424,"processing_time_ms":7425,"cost_usd":7426},7921,645,3574,0.00294775,{"type":14,"value":7428,"toc":7485},[7429,7433,7440,7444,7447,7461,7465],[17,7430,7432],{"id":7431},"shift-from-benchmarks-to-production-testing","Shift from Benchmarks to Production Testing",[22,7434,7435,7436,7439],{},"Most public discourse on AI evaluation is stuck in the 'chatbot era,' focusing on static datasets that break whenever a model or tool harness changes. Instead of chasing benchmark scores, treat agent evaluations like standard software tests (unit and end-to-end). Use tools like ",[26,7437,7438],{},"vitest"," or similar harnesses to run tests locally as code. This approach is more resilient to the rapid pace of change in the AI ecosystem and avoids the 'eval theater' that delays product shipping.",[17,7441,7443],{"id":7442},"the-floor-raising-framework","The 'Floor Raising' Framework",[22,7445,7446],{},"When building agents, distinguish between the 'ceiling' (the agent's peak capability) and the 'floor' (the worst thing it can do, such as leaking data or hallucinating harmful actions). User trust is built by raising the floor, not just chasing capability. To effectively manage this, you must track two specific metrics for every issue:",[7094,7448,7449,7455],{},[41,7450,7451,7454],{},[44,7452,7453],{},"Temporal Origin:"," When did this issue start? Knowing if an issue is new helps you correlate it with recent code or model changes.",[41,7456,7457,7460],{},[44,7458,7459],{},"User Impact:"," What percentage of users are affected? This allows you to prioritize fixes based on scale rather than reacting to every single trace.",[17,7462,7464],{"id":7463},"tactical-lessons-for-issue-discovery","Tactical Lessons for Issue Discovery",[38,7466,7467,7473,7479],{},[41,7468,7469,7472],{},[44,7470,7471],{},"Avoid Naive Clustering:"," While clustering traces is useful for one-off analysis, it fails at scale. It lacks temporal context and often groups issues with different root causes together, making it difficult to track regressions over time.",[41,7474,7475,7478],{},[44,7476,7477],{},"Use Code-Based Classifiers:"," Apply the 'code mode' pattern to your traces. Write deterministic classifiers and run them in sandboxes at production scale. This provides more control than automated clustering.",[41,7480,7481,7484],{},[44,7482,7483],{},"Don't Ask Agents to Find Anomalies:"," Agents are poor at detecting anomalies. Instead, use deterministic methods (like tracking keyword frequency spikes) to identify potential issues, then use the agent to investigate those specific, tractable anomalies.",{"title":90,"searchDepth":91,"depth":91,"links":7486},[7487,7488,7489],{"id":7431,"depth":91,"text":7432},{"id":7442,"depth":91,"text":7443},{"id":7463,"depth":91,"text":7464},[97],{"content_references":7492,"triage":7502},[7493,7496,7499],{"type":110,"title":7494,"url":7495,"context":114},"Workshop","https:\u002F\u002Fgithub.com\u002Fraindrop-ai\u002Fworkshop",{"type":110,"title":7497,"url":7498,"context":114},"how to eval","https:\u002F\u002Fhowtoeval.com\u002F",{"type":110,"title":7500,"url":7501,"context":112},"Sentry","https:\u002F\u002Fsentry.io\u002F",{"relevance":7165,"novelty":116,"quality":116,"actionability":116,"composite":7254,"reasoning":7503},"Category: AI & LLMs. The article provides a practical framework for evaluating AI agents, addressing the audience's pain point of moving from theoretical benchmarks to actionable production testing. It offers specific metrics and tactical lessons that can be directly applied to improve AI product reliability.","\u002Fsummaries\u002F5bc49edfb5f3bc68-raising-the-floor-practical-ai-agent-evaluation-summary","2026-08-12 18:00:34","2026-08-13 03:24:46",{"title":7421,"description":90},{"loc":7504},"5bc49edfb5f3bc68","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=jHMiYtjoJfA","summaries\u002F5bc49edfb5f3bc68-raising-the-floor-practical-ai-agent-evaluation-summary",[7513,7514,135,134],"agents","ai-tools","Stop chasing benchmark scores and start treating agent evaluations like production software tests. Focus on identifying when issues start and their impact on user volume to build reliable, trust-based AI products.","This is a candid talk from the CTO of [Raindrop](https:\u002F\u002Fraindrop.ai) arguing that most standard AI evaluation practices are \"theater\" that fail to account for the unpredictable nature of real-world agents. He advocates for prioritizing \"floor-raising\" (preventing catastrophic failures) over chasing benchmarks and points to his company's resources, [Workshop](https:\u002F\u002Fgithub.com\u002Fraindrop-ai\u002Fworkshop) and [howtoeval.com](https:\u002F\u002Fhowtoeval.com), as practical alternatives for production monitoring.",[135,134],"7dXm_xMJS_DwDkxpJNBFcpCLF4NzQdSaT9ejH9lyXYA"]