While meditation works for some people, I think that the narrative of meditation will work for everyone if you just stick with it is false. I don't have rigorous data to support this; I only have my only experiences and observations. Some people take to it very easily. I gave meditation an attempt multiple times during different periods of my life (joining meditation events, short classes for an hour or two each week, yoga teacher training, and lots of shorter mediations as part of yoga classes). I never really got anything out of it. I honestly don't think that it benefited me at all.

So my non-expert, layperson's perspective would be this: just because mediation is a useful tool for some people, doesn't mean it is going to be the right tool for you and for what you want to do. If you want increased calm, self-awareness  and wellbeing, there is more than one path to get that.

Context: I have been a practicing Buddhist, primarily in the Plum Village Tradition, since 2011. In addition to my personal practice, I am currently training to become a "lay monastic" (essentially, just doing more formal Sangha-building).

First and foremost, I appreciate your openness about your experience. I definitely understand how frustrating this can be, and you are certainly not alone in this. I feel comfortable in assessing that most, if not all, struggle with this to some extent - even years into practice. Myself included. There are so many variables that impact one's ability to be present and mindful, and it is important to be compassionate toward yourself whenever this happens for you. 

Broadly, what has helped me in my own practice is not conceptualizing meditation as meditation = sitting meditation. There are many forms of meditation (walking, deep relaxation/resting, hugging, eating, etc.) and every moment is an opportunity to be mindful and present. Another aspect of this particular tradition that I have found helpful is to move away from conceptualizing meditation and mindfulness as means to an end (consequentialist/utilitarian-esque thinking), and rather, as the end in itself (this is largely how it is regarded in my main tradition, but others regard meditation as [TLDR:] the way to enlightenment). 

I have also found that more diligently studying the sutras/teachings/texts/history has deepened my understanding and helped my practice, but I have simultaneously worked to avoid over-intellectualizing; instead, working to embody the practices and to integrate them into my daily life (incrementally over time, even if it is just a few seconds of being more mindful per day/week/month). 

As for jumping into a retreat - results vary drastically. Some find this beneficial, others find it extremely challenging and occasionally harmful (the latter is often true if people are struggling with deep emotional/psychological topics). I tend to encourage people to try a one-day retreat, maybe a weekend, rather than going all in on a week or more. It may also be worth considering which type of retreat you partake in; I would recommend doing research and tuning into what sounds best based on your wants/needs at the time you are searching. I attend retreats in various traditions throughout the year, but I most often attend retreats at Plum Village centers, as I have found that style most suitable for my wants/needs. I do find that other retreats/traditions can be more helpful at different points.

It is difficult to discuss this at length in a comment format, but I would be happy to hear more about your experience and discuss it further, if you think doing so would be beneficial (others are welcome, too). Feel free to schedule some time for us to chat.

In the interim, I am happy to answer questions here, in a message, or by email.

Like Joseph says, conventional meditation doesn't work for everyone. Don't force yourself to try and do it. It doesn't work well for me either. Maybe less conventional forms of meditation would work for you: walking meditations, meditations targeted towards neurodiverse people (which is more literal) or just your own interpretation/take. I personally prefer focusing on mindfulness more broadly than meditation. And in my experience, cardio exercises and isolating myself to enjoy music are clearly more effective mindfulness exercices than meditation ever was for me.

Disclaimer: I have aphantasia and it seems that my subjective conscious experience is far from usual.^[1] I don't have deep meditation experience; I have meditated a cumulative 300+ hours since 2015. I have never meditated for more than 2 hours a day.

I've found Headspace's mindfulness stuff unhelpful, albeit pleasant. It was just not what I needed but I only figured it out after a year or so. Metta (loving-kindness) is the practice I consistently benefit from most, also for my attention and focus. It's the best "un-clencher" when I'm by myself. And it can get me into ecstatic states or cessation. Especially "open-monitoring" metta is great for me.

Related writing that resonates with me and has shaped my perspective and practice are:

1. Charlie Rogers on self-love.
2. Nick Cammarata on attachment/"clenching", meditation as a laxative and how that affects happiness, clarity and motivation.
3. A lot of the Qualia Research Institute's work is tangentially fascinating. They summarize the neuroscience on meditation quite well (afaict) and their scales of pain and pleasure map onto my experiences, too.

Maybe some of this can help you identify your own path forward?

I have a friend who did a personalized retreat with a teacher for 3 days and made major breakthroughs; i.e. overcoming multi-year depression, getting to 6th Jhana. The usual retreats are probably inefficient, it seems better to have a close relationship and tighter feedback loops with a teacher.

1. ^{^}

   I don't have an inner voice, I don't have mental imagery, my long-term memory is purely semantic (not sensory or episodic) and I have little active recall capacity. Essentially, my conscious mind is exceptionally empty as soon as I reduce external input. That doesn't mean I'm naturally great at focusing (there's so much input!). I'm just not fussed about things for longer than a few minutes because my attention is "stuck" in the present. I forget most things unless I build good scaffolds. I don't think this architecture is better or worse - there are tradeoffs everywhere. Happy to talk more about this if it seems relevant to anyone.

From my knowledge of Tibetan Buddhism, and the advice I have received in this context, the goal of meditation is not to increase well-being. I know there is evidence out there about this, but given that this is not the goal of meditation, perhaps there is something to the old wisdom? And I would guess the evidence shows an average improvement over many people in well being, but not that every single person improved. I would not be shocked if for a few people it even decreased well being (that is certainly what I understand from Tibetan Buddhism - it is said that sometime meditation makes you process some bad karma and that seems to not feel good for people). If your goal is increased well-being, I would rather try any and all things with evidence of improved well-being to see if something fits you. And why not try something where evidence is not yet produced? If you have time it can't hurt and might be a fun mission to discover more about life and how different people live their lives. The type of meditation I know is focusing on your breath in case that is helpful - there are so many types of meditation out there!

My understanding is that to get most of the benefits of meditation you don't have to meditate for longer than ten minutes. I could be full of shit here, would very much be interested in evidence to the contrary.