You seem to be misunderstanding the meaning of attempt

To attempt something does not means you have to be able to succeed. A character with a Strength of 2 can attempt to kick down a door with a strength DC of 25. They will fail but they can still attempt it. Nothing makes the act of trying impossible, nor is it so absurd that a roll is pointless once other people or equipment is involved. (If four people are attempting to lift a crate together, each person is not attempting to lift the full weight of the crate; each person is trying to lift roughly 1/4th of the weight of the crate.)

The DMG (pg 237) gives an example of what it considers checks that are not worth using dice to adjudicate:

Is a task so easy and so free of conflict and stress that there should be no chance of failure? Is a task so inappropriate or impossible - such as hitting the moon with an arrow - that it can't work?

It it not enough that the task be impossible, it need to be ludicrously impossible make attempting pointless.

It also helps if you have the full quote from the Player's Handbook on working together:

A character can only provide help if the task is one that he or she could attempt alone. For example, trying to open a lock requires proficiency with thieves' tools, so a character who lacks that proficiency can't help another character in that task.

Based the examples, something you can attempt is not the same as something they can succeed at. Something they cannot attempt is something that by the rules they cannot even make the check in the first place, such as lacking the skill or being impossible to even try to do. Four people could lift a rock; four people cannot lift a mountain.

I can attempt to pick a lock that I cannot succeed in picking; I cannot attempt to pick a lock that is in a different building from the one I am standing in, or a lock with no tools whatsoever.

Group checks

In any case many of these would be group checks, that's a different mechanism. Pg 176 of the PH, a task in which some characters can make up for the deficiencies of others.

When a number of individuals are trying to accomplish something as a group, lhe DM might ask for a group ability check...Group checks don't come up very often. and they're most useful when ali lhe characters succeed or fail as a group.

An example I like to use is if you had 20 people rowing in a ship, no one person can move the ship, but if one person fails the ship still moves so it is a perfect example of a group check.

By RAW, you're right

You've cited the relevant rule already. If a character doesn't have the Strength to lift/drag the boulder (see Lifting and Carrying, PHB p.176), there's no way they're going to get it moving, and that falls into the 'debatable' category of being able to attempt the action, at best.

But clearly that makes no sense

Lifting or pushing a large object in tandem with others is the poster child for tasks that benefit from multiple participants (so long as there's room for everyone).

You (assuming you're the DM in this scenario; if you're not, pitch this to them) would be well within your rights to rule that anything with a Strength score would be able to use the Help action to contribute their excess Lift/Drag capacity to the task. You could also, as detailed in this answer, just use raw Strength scores, though you'd have to make a judgment call on what the threshold would be in either case.

So how might you adjudicate this action?

Effectively, your players would be creating an improvised Rolling Sphere trap, from which we can borrow mechanics:

... a 10 foot diameter rolling sphere of solid stone ... moves 60 feet in a straight line. [It] can move through creatures' spaces, and creatures can move through its space, treating it as difficult terrain.

Whenever the sphere enters a creature's space or a creature enters its space while it's rolling, that creature must succeed on a DC 15 Dexterity saving throw or take 55 (10d10) bludgeoning damage and be knocked prone.

... As an action, a creature within 5 feet of the sphere can attempt to slow it down with a DC 20 Strength check. On a successful check, the sphere's speed is reduced by 15 feet. If the sphere's speed drops to 0, it stops moving and is no longer a threat.

[DMG p.123]

But first they would need to actually get the boulder rolling, which would require enough of the party to spend an action trying to move the boulder (pooling their strength as stated above, until they had enough to move the boulder).

My Solution

My own solution is to add a caveat to the the help action:

If the creature with the highest ability modifier can not complete the task alone, then the help action becomes a "group effort", becoming a group check without advantage.

How?

Each participating character performs a Help action, but at the end of the round a group check is performed to see if the task is performed. The GM should therefore adjust the DC according to who is participating and how many are participating.

The rules concerning Group actions can be found in PHB p175

GROUP CHECKS

When a number of individuals are trying to accomplish something as a group, the DM might ask for a group ability check. In such a
situation, the characters who are skilled at a particular task help
cover those who aren't.

To make a group ability check, everyone in the group makes the ability check. If at least half the group succeeds, the whole group
succeeds. Otherwise, the group fails.

The DM should determine the difficulty check depending on type of challenge and number of people trying to do it. For example lifting the same boulder might be DC20 if 4 people attempt it, but DC 10 if 10 people try to lift. While scrambling up a wall in a timely manner might be a flat DC15 regardless of number of people, with the stronger/luckier ones helping the weaker/unluckier ones.

Why?

• A group check would emphasize the task is a team effort.


• No advantage is given to distinguish the it from a regular Help action. This makes it easier to understand and removes confusion.


• The task is performed at the end of the round so as to allow other players to join in, while also allowing some enemies to try and stop the plan. This is done for simplicity sake, as an arbitrary action or reaction would be too convoluted. This can be explained in-game as the time required to coordinate the task.

Why the roll?

Other answers mention pooling together modifiers which would then succeed a threshold. While effective, it isn't as tactical. If enough characters contribute, then the plan always succeeds.

This is where the aforementioned group check comes into play. Players must judge the risk of contributing . Perhaps too many characters contribute and they overcompensate, wasting actions that could have been spent more wisely. Or, perhaps a plan fails and a player regrets not contributing to the effort.

When is it a "Group Effort"?

If the task can possibly be performed by a participating member, then it is regular Help assist. Else, if no participating member can perform the task, it becomes a "Group Effort"

Example 1

A Goliath and a Dwarf want to ram a spear through a Dragon's heart. The GM ask themselves whether the Giant or the Dwarf would have enough strength to do it alone. If at least one of them can feasibly do it alone, then it becomes a regular Help assist. Else, if both our incapable of doing it alone, then it becomes a "Group Effort".

The GM states that the task would be group effort. The dwarf moves, reacts and takes the Help action on its turn. The Goliath does the same. On the end of the round a group strength check is made against a DC of say 20. If an hulking orc joins in the effort, the DC is reduced to say 10. If a tiny gnome joins in, then the DC is reduced to say 15. If both the orc and the gnome join in the effort the DC is reduced even further so say 5.

Finally a group strength check is performed. If it passes, the spear manages to pierce the dragons heart. If the group fails to pass, then the spear bounces of the Dragon's scales.

Example 2

Three mages want to send glass shards hurtling at their enemies. One might conjure sand, the other fire and the last one air. Since the action is impossible to perform alone, this becomes a group effort.

Each mage uses it's action to Help. At the end of the round, they pass a group intelligence check. If they pass, then they skewer their enemies with a thousand glasses pieces. If they fail to pass the intelligence check, then the contributing magic spells are misaligned.

Situation 1: Don't let them roll.

There are rules on how much weight characters can push or carry, in the PHB on page 176.

Carrying Capacity. Your carrying capacity is your Strength score multiplied by 15. This is the weight (in pounds) that you can carry, which is high enough that most characters don’t usually have to worry about it.

Push, Drag, or Lift. You can push, drag, or lift a weight in pounds up to twice your carrying capacity (or 30 times your Strength score). While pushing or dragging weight in excess of your carrying capacity, your speed drops to 5 feet.

Size and Strength. Larger creatures can bear more weight, whereas Tiny creatures can carry less. For each size category above Medium, double the creature's carrying capacity and the amount it can push, drag, or lift. For a Tiny creature, halve these weights.

Use those rules to determine whether the characters can move or throw the boulder or not. Since throwing is significantly harder than carrying, you could halve the carrying capacity.

If you prefer rolling, you will have to homebrew, for example by letting the entire group make one roll and adding the STR modifiers of all participants (disclaimer: I have not tested this homebrew). How high you set the DC is up to you.

Situation 2: for other checks, such as climbing a high wall, it depends on how your players approach the challenge.

In the example of climbing a wall, the characters might attempt to get up by giving climbing on each other. In this case, you could use the carrying rules as well, since the guy at the bottom effectively has to carry everybody above him (and the second guy everybody above him, etc.). This would be 30 times STR, since it would count as "lifting". Considering that he won't be moving, though, you could, in addition, allow STR checks if the carried creatures' weight exceeds the lifting capacity of the carrier.

Do note that your players could attempt a human pyramid, in which case distributing the weight would be a little more complicated. In either case, carrying more than one person on your shoulders is not easy, so the fact that they might be too weak for a human pyramid is not too illogical.

If your players try to "run up the wall", with one character pushing them upwards and giving them a raised foothold to jump off, you could resolve this using the High Jump rules in the PHB on page 182:

High Jump. When you make a high jump, you leap into the air a number of feet equal to 3 + your Strength modifier if you move at least 10 feet on foot immediately before the jump. When you make a standing high jump, you can jump only half that distance. Either way, each foot you clear on the jump costs a foot of movement. In some circumstances, your DM might allow you to make a Strength (Athletics) check to jump higher than you normally can.

You can extend your arms half your height above yourself during the jump. Thus, you can reach above you a distance equal to the height of the jump plus 1.5 times your height.

Since they're jumping off another character, you can add whatever height they jump off the character to the jump height, as well as the lifting character's strength score in feet, if he's pushing the jumper upwards.